DF-4327 — sf(4) sf_stop() callout_stop() vs sf_stats_update() AB-BA deadlock

File: sys/dev/netif/sf/if_sf.c:1344 (callout_stop under serializer) sys/dev/netif/sf/if_sf.c:1396 (callback blocks on same serializer)

sf_stop() is always entered with the driver serializer held (ifconfig down, detach, watchdog, abnormal-IRQ reinit) and calls the BLOCKING callout_stop(). If the 1 Hz stat timer has fired on another CPU, sf_stats_update() is parked in lwkt_serialize_enter() on that same serializer; callout_stop() sleeps on "costp" (kern_timeout.c:923) waiting for the callback; the callback waits for the serializer held by the stopper: permanent AB-BA wedge (DF-3715 family).

Build / Run (guest)

cd /root/kmod/sfcal && make SYSDIR=/usr/src/sys && kldload ./sf_callout_harness.ko
sysctl kern.sfcal.arm=1 ; sleep 3
(sysctl kern.sfcal.stop=1 &) ; sleep 10
ps -o pid,stat,wchan,command -ax | grep sfcal.stop

Expected (run.log)

sfcal[stop]: callback fired=10 ran=9 -> BLOCKED on serializer. calling callout_stop() ...
<thread> D4 costp sysctl kern.sfcal.stop=1     # NEVER returns

No "callout_stop returned" line is ever printed; the sysctl thread and a softclock thread are wedged permanently. Guest must be reset afterwards.

DF-4327 — verdict: REPRODUCED (protocol replica on live kernel)

Root cause (path:line)

• sys/dev/netif/sf/if_sf.c:1344 — callout_stop(&sc->sf_stat_timer); inside sf_stop(), which every caller enters holding ifp->if_serializer (sf_ioctl SIOCSIFFLAGS down:525, sf_detach:798, sf_watchdog:1436 via sf_stop+serializer, sf_init from sf_intr:1090 with the ISR holding it).
• sys/dev/netif/sf/if_sf.c:1396 — the callback sf_stats_update() begins with blocking lwkt_serialize_enter(ifp->if_serializer) and re-arms itself at :1421.
• sys/kern/kern_timeout.c:1091 — callout_stop() = _callout_cancel_or_stop( cc, CALLOUT_STOP, 1): with sync=1 it ssleep("costp")s (kern_timeout.c:923) until an INPROG callback completes. => stopper holds serializer + waits "costp" for callback; callback waits for serializer: AB-BA, permanent. Mechanism identical to DF-3715 (em(4)), proven there on this same kernel; here tied to sf's exact call pattern.

What was proven on the guest (run.log)

• Replica module reproduces the exact protocol: timer callback that (1) lwkt_serialize_enter's, (2) re-arms under the serializer; a stopper that holds the serializer, catches the callback provably blocked (fired=10 ran=9), then calls callout_stop().
• Result: the stop thread sits in state D4, wchan costp forever: 979 D4 costp sysctl kern.sfcal.stop=1 and no "callout_stop returned" line ever prints. One softclock thread is simultaneously parked on the serializer. Guest had to be reset.

Impact

Local/trigger-conditional permanent hang (DoS): wedges the ifnet serializer (anything touching sf0 hangs) and one CPU's softclock thread (all callouts on that CPU stall). Triggers: ifconfig sf0 down / driver detach / abnormal interrupt reinit racing the 1s stat timer — root/admin-initiated or event-driven, timing window each hz tick. Not memory corruption.

Fix

callout_stop_async() under the serializer + !IFF_RUNNING guard in the callback + callout_drain() in sf_detach after releasing the serializer (fix.diff).

Fix validation

fix_status=not_testable: fix lives in if_sf.c (hardware driver); the replica module cannot observe sys/ changes. The fix pattern (stop_async + guard + drain) is the standard DFly idiom for this proven family.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.