DF-4064 — fxp(4) fxp_stat_timer AB-BA vs fxp_stop (callout subsystem deadlock)

What

fxp_tick (if_fxp.c:1441) is a non-MPSAFE callout (plain callout_init, if_fxp.c:399) whose callback blocks in lwkt_serialize_enter(if_serializer) (if_fxp.c:1449) and re-arms itself inside the serializer (if_fxp.c:1552). fxp_stop() tears it down with callout_stop(&sc->fxp_stat_timer) (if_fxp.c:1577) while holding the serializer (ASSERT_SERIALIZED :1568; reachable from fxp_detach :738, fxp_shutdown :787, fxp_suspend :805, fxp_init :1651, fxp_ioctl SIOCSIFFLAGS :2078, fxp_watchdog via fxp_init).

DragonFly's callout_stop() is synchronous (kern_timeout.c:1091 → _callout_cancel_or_stop sync=1; ssleep "costp" at :920) — it waits for an in-progress callback. A callback dequeued and parked on the serializer therefore deadlocks against the stopper: stopper sleeps in callout_stop holding the serializer; softclock sleeps in lwkt_serialize_enter wanting it. Because the callout is non-MPSAFE, the parked softclock thread also holds the MP lock (kern_timeout.c:575-580), widening the wedge system-wide.

Build / run / expected

1. Driver-level attempt (real hardware): boot the guest with the temp launcher dfbsd-qemu/vm_fxp.sh (adds -device i82557b,netdev=n1; vm.sh untouched) — fxp0 attaches for real (device fxp is in X86_64_GENERIC). Run fxp_race*.sh (up/down hammers + state-dump storms). Expected: a permanent wedge if the 1 Hz tick fires while a non-stop-holder holds the serializer. Observed: no wedge — structurally unreachable via ifconfig on this kernel (see VERDICT.md: the timer is armed from the fxp netisr CPU, so it only ever runs when that CPU's ifnet queue is drained, i.e. the serializer is free).
2. Deterministic protocol replica (the proof): cd /root/fxpkmod && make # fxp_callout_replica.ko (-Werror clean) kldload /root/fxpkmod/fxp_callout_replica.ko sysctl kern.fxr.arm=1 sysctl kern.fxr.stop_blk=1 # fxp_stop shape Expected (serial console): fxr: handler BLOCKED on our serializer … then fxr: IF THIS IS THE LAST fxr LINE, AB-BA DEADLOCK IS PROVEN … and nothing after it — callout_stop() never returns. ps on a second ssh: stopper D0 costp, softclock 5 B5 slize. Observed: exactly that (run.log, guest_pre_reset.txt).
3. Fixed shape (fix.diff protocol): cd /root/fxpkmod_fixed && make # fxp_callout_replica_fixed.ko kldload /root/fxpkmod_fixed/fxp_callout_replica_fixed.ko sysctl kern.fxrf.arm=1 sysctl kern.fxrf.stop_blk=1 # same hold + callout_stop kldunload fxp_callout_replica_fixed Expected: fxrf: callout_stop RETURNED cleanly, pending=0 + clean unload. Observed: exactly that (run.patched.log).

After any wedged run: vm.sh reset with-src.

DF-4064 — VERDICT

Bottom line

Mechanism PROVEN on the guest kernel by deterministic replica of if_fxp.c's exact callout/serializer protocol; fxp-specific runtime hardware/timing-gated (inconclusive per family convention — same standard as em(4) DF-3715 and emx(4) DF-3738).

The code shape (source-certain)

• fxp_stat_timer is a non-MPSAFE callout: plain callout_init at sys/dev/netif/fxp/if_fxp.c:399 (vs em/emx which use callout_init_mp — the fxp variant additionally runs the callback under the MP lock, sys/kern/kern_timeout.c:575-580).
• fxp_tick (if_fxp.c:1441-1555) blocks in lwkt_serialize_enter(sc->arpcom.ac_if.if_serializer) at :1449, runs its whole body under it, and self re-arms inside it at :1552 (callout_reset(&sc->fxp_stat_timer, hz, fxp_tick, sc)).
• fxp_stop calls callout_stop(&sc->fxp_stat_timer) at :1577 while holding the serializer (ASSERT_SERIALIZED :1568). Callers: fxp_detach :738, fxp_shutdown :787, fxp_suspend :805, fxp_init :1651 (thus every ifconfig fxp0 up, every SIOCSIFFLAGS re-init, and the watchdog path fxp_watchdog :1632), fxp_ioctl SIOCSIFFLAGS down :2078.
• DragonFly callout_stop is synchronous: kern_timeout.c:1091 → _callout_cancel_or_stop(cc, CALLOUT_STOP, 1); ssleep(c, &c->spin, 0, "costp", 0) at kern_timeout.c:920 while the callback is INPROG.
• lwkt_serialize_enter blocks: atomic_intr_cond_enter(..., lwkt_serialize_sleep, ...); tsleep(s, PINTERLOCKED, "slize", 0) at lwkt_serialize.c:227-235.

A callback that has been dequeued (INPROG) and parked on the serializer deadlocks against any stopper (AB-BA); both threads wedge permanently, the affected CPU's softclock stops delivering callouts, and — fxp-only twist — the parked callback holds the MP lock, stalling every BGL-dependent path system-wide.

Proven on the guest (replica, deterministic)

fxp_callout_replica.ko replicates the protocol exactly (non-MP callout_init, blocking lwkt_serialize_enter at the fxp_tick:1449 position, hz self re-arm at the :1552 position, callout_stop under the serializer at the :1577 position). Run transcript (run.log / console):

fxr: armed (fxp_stat_timer shape, hz period)
fxr: handler BLOCKED on our serializer (fired=28 ran=27) - entering callout_stop() [fxp_stop:1577 shape]
fxr: IF THIS IS THE LAST fxr LINE, AB-BA DEADLOCK IS PROVEN ...
<nothing — callout_stop never returned>

ps on the wedged guest (guest_pre_reset.txt):

-1 B5     slize    softclock 5     <- parked in fxp_tick's lwkt_serialize_enter
231052 D0     costp    sysctl           <- fxp_stop shape, stuck in callout_stop

100% deterministic (armed handler parks the moment the stopper-side holder sleeps holding the serializer; callout_stop then waits forever).

Driver-level attempt on real emulated fxp0 (honest negative)

fxp0 was attached for real on the guest by booting QEMU with a temp launcher adding -device i82557b,netdev=n1 (vm_fxp.sh; vm.sh untouched; device fxp is compiled into X86_64_GENERIC:233):

fxp0: <Intel 82557 Pro/100 Ethernet> port 0xc080-0xc0bf irq 11 at device 4.0 on pci0
miibus0: <MII bus> on fxp0

Four adversarial hammer campaigns were run (~10 minutes total: 4-way up/down loops; phased arm/hold/stop cycles; 6-way saturating state-dump storms; quiet-period diagnostics). The tick demonstrably fires ("fxp0: device timeout" via the rx_idle → fxp_mc_setup → watchdog chain appeared on the console exactly as the code predicts), but the wedge was never reached, and the reason is structural on this kernel:

• the timer is always armed from the fxp netisr CPU (fxp_init runs in the ioctl netmsg on the ifnet's serializer CPU), so the callback fires on that same CPU's softclock;
• the tick's only blocking point is its lwkt_serialize_enter entry (:1449); softclock on that CPU only gets to run when the netisr queue is drained — at which moment the serializer is free, so the enter succeeds immediately and the callback never parks;
• no other path sleeps while holding the ifnet serializer before issuing a callout_stop (fxp_stop's callout_stop is the first thing it does, before any long-held work — unlike em(4)/emx(4) whose stop does long work around the stop), so the em-family interleaving cannot be reconstructed with userland ifconfig.

Conclusion at the driver level: the hazard is carried in the code (identical shape to the proven-family bugs; any future change that arms the callout from another CPU, sleeps holding the serializer, or splits fxp_stop re-opens the deadlock), but on the current kernel + driver the trigger is hardware/timing-gated beyond what ifconfig storming reaches — hence inconclusive per family convention.

Fix validation (protocol level, same guest)

fxp_callout_replica_fixed.ko implements the exact fix.diff protocol (lwkt_serialize_try() + 1-tick callout_reset retry — the emx_txgc_timer pattern validated for DF-3738). Same fxp_stop-shaped stopper holding the serializer for 3 s:

fxrf: armed (FIXED fxp_tick shape)
fxrf: holding serializer, callout_stop() [fxp_stop shape]...
fxrf: callout_stop RETURNED cleanly, pending=0
kldunload fxp_callout_replica_fixed -> FIXED_SHAPE_OK

Baseline shape = permanent wedge; fixed shape = clean stop + clean unload. Full driver-kernel rebuild validation not performed (module-level protocol proof, family standard).

Artifacts

• run.log — baseline replica transcript + ps evidence + driver-level context
• run.patched.log — fixed-shape transcript
• build.log / build.fixed.log — full untrimmed -Werror-clean guest builds
• boot-fxp-full.log — complete serial console of the driver-level session (fxp0 attach, race campaigns, DMA-timeout flood, replica wedge lines)
• guest_pre_reset.txt — full ps + logs pulled off the wedged guest
• fxp_race*.sh — the four driver-level hammer campaigns
• vm_fxp.sh — temp QEMU launcher (vm.sh untouched)
• fix.diff — try-enter + 1-tick retry for fxp_tick

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.