DF-4275 — epic_stop() callout_stop AB-BA deadlock (if_tx.c)

Finding: sys/dev/netif/tx/if_tx.c — SMC EtherPower II (EPIC/100) driver. DF-3715 family instance (same mechanism as the em(4) finding, this file was missed in pass 1).

epic_stats_update()                epic_ifioctl()/epic_ifwatchdog()/epic_intr()/epic_detach()
(if_tx.c:796-811)                  ------------------------------------------
lwkt_serialize_enter(              lwkt_serialize_enter(ifp->if_serializer);
    ifp->if_serializer);   <--+    epic_stop(sc);                 (if_tx.c:1341)
    mii_tick(mii);             |   callout_stop(&sc->tx_stat_timer);
    callout_reset(...);        |       |
lwkt_serialize_exit();       blocked  |  waits 'costp' for the callback
                                   <--+  to finish (kern_timeout.c:888-921)

In this kernel callout_stop() is fully synchronous (sys/kern/kern_timeout.c:1091-1094 -> _callout_cancel_or_stop(..., sync=1) -> ssleep(c, &c->spin, 0, "costp", 0) at kern_timeout.c:920). If the 1-second mii tick has been dequeued by softclock (CALLOUT_INPROG set, kern_timeout.c:567) and is parked in lwkt_serialize_enter() on the interface serializer — which is its normal state whenever the serializer is contended, e.g. under RX interrupt load, inside epic_stop_activity()'s millisecond DELAY loops, or during a concurrent ifconfig — then any thread that enters epic_stop() while holding ifp->if_serializer blocks forever waiting for that callback.

Entry points holding ifp->if_serializer across epic_stop()->callout_stop(): - epic_ifioctl SIOCSIFMTU (if_tx.c:370-373: epic_stop+epic_init) - epic_ifioctl SIOCSIFFLAGS down (if_tx.c:389-391) - epic_ifwatchdog (if_tx.c:781-782: epic_stop+epic_init; watchdog itself runs under ifnet_tryserialize_all from if_slowtimo_dispatch, net/if.c — wedging it also wedges netisr0) - epic_intr PCI-fatal (if_tx.c:702-703) - epic_detach (if_tx.c:300-304)

Because if_tx.c uses plain callout_init() (if_tx.c:200, non-MPSAFE), the stuck softclock thread also holds the MP lock, so the wedge degrades the whole system, not just one interface.

Reproduction

The NIC is hardware-gated (no SMC83C170 in the QEMU guest), but the deadlock is a callout-core/serializer-discipline mechanism: a protocol replica module reproduces the exact lock discipline (tick callback that first enters the serializer; stopper holding the serializer calling synchronous callout_stop) and demonstrates the permanent 'costp' sleep.

./build.sh     # df4275rep.ko
./run.sh       # kldload; expect stopper parked forever in "costp"

Expected output (console/dmesg):

df4275: tick() dequeued; entering serializer (blocks: stopper holds it)
df4275: stopper HOLDS serializer; tick() INPROG+blocked; calling callout_stop() ...
df4275: stopper_td=... wchan=... wmesg="costp"
df4275: RESULT: DEADLOCK REPRODUCED -- stopper permanently asleep in "costp"

Artifacts

• df4275_rep.c — replica module source
• build.log — module build output
• run.log — console capture of decisive run
• verdict.json — machine verdict

DF-4275 — epic_stop()/epic_stats_update() AB-BA deadlock (sys/dev/netif/tx/if_tx.c)

VERDICT: REPRODUCED (mechanism), impact = permanent kernel deadlock / DoS.

What the finding claims

if_tx.c re-arms a 1-second callout in epic_stats_update() (if_tx.c:808) whose callback FIRST does lwkt_serialize_enter(ifp->if_serializer) (if_tx.c:803), while every path that stops the interface holds that same serializer across epic_stop() -> callout_stop(&sc->tx_stat_timer) (if_tx.c:1341):

• epic_ifioctl SIOCSIFMTU if_tx.c:370-373 (epic_stop + epic_init)
• epic_ifioctl SIOCSIFFLAGS if_tx.c:389-391 (interface down)
• epic_ifwatchdog if_tx.c:781-782 (runs under ifnet_tryserialize_all from if_slowtimo_dispatch, net/if.c — a wedge here also wedges netisr0)
• epic_intr PCI-fatal if_tx.c:702-703
• epic_detach if_tx.c:300-304

In this kernel callout_stop() is fully synchronous (kern_timeout.c:1091-1094 -> _callout_cancel_or_stop(..., sync=1) -> ssleep(c, &c->spin, 0, "costp", 0), kern_timeout.c:920) and waits for a callback that is CALLOUT_INPROG (dequeued, kern_timeout.c:567). If the tick has been dequeued and is parked in lwkt_serialize_enter() on the interface serializer — its normal state whenever the serializer is contended (RX interrupt load, epic_stop_activity()'s multi-millisecond DELAY loops at if_tx.c:1238-1243, any concurrent ioctl) — then the stopper waits forever for a callback that is waiting forever for the serializer the stopper holds. Because if_tx.c uses plain callout_init() (if_tx.c:200, non-MPSAFE), the wedged softclock thread also holds the MP lock, degrading the whole system.

How it was verified

The SMC83C170 is not emulated by QEMU, so the driver-level trigger cannot run on the guest. The deadlock, however, is a pure callout-core/serializer discipline; a protocol-replica KLD (df4275_rep.c) reproduces the exact lock order: a stopper kthread takes the serializer, waits until the 1-tick callback has been dequeued and parked on it, then calls callout_stop() — precisely what any serialized ioctl path does when it calls epic_stop() while the mii tick is blocked.

Result on DragonFly 6.5-DEVELOPMENT #1 (kernel rebuilt from the audit tree, 2026-09-19), guest pid 3641097 cycle (run.log):

df4275: tick() dequeued; entering serializer (blocks: stopper holds it) df4275: stopper HOLDS serializer; tick() INPROG+blocked; calling callout_stop() ... df4275: stopper_td=0xfffff8008ed96600 wchan=... wmesg="costp" (x5, 1s apart) df4275: RESULT: DEADLOCK REPRODUCED ...

ps -axo ...,wchan,comm: -1 B5 costp df4275stop <- stopper parked in callout_stop's wait -1 B5 slize softclock 5 <- softclock blocked in lwkt_serialize_sleep INSIDE the callback (holding MP lock)

Both sides of the AB-BA are directly observable in the process table; the state persists indefinitely (5+ observation seconds, and remains after the observing shell exits). This is the identical mechanism proven for DF-3715 (em(4)); if_tx.c was missed in pass 1.

Two earlier replica iterations (kept for honesty): 1. stopper waited for tick-inprogress via tsleep(...,1) — the deadlock manifested one step earlier: the stopper's sleep timeout is delivered by the very softclock wheel the tick wedges, so the stopper parked forever in "df4275w" (run captured 03:0x). Valid, but not the canonical signature. 2. stopper busy-waited with DELAY() — LWKT kernel threads of equal priority are not preempted, so the spin starved softclock on the same CPU and the callback never ran (no deadlock; replica bug, fixed with lwkt_yield()).

The final replica (yield-wait) produces the canonical 'costp' wedge.

Environment note (not a finding): modules built from the CURRENT /usr/src against the SHIPPED guest kernel (#0, Jul 2 2026) GPF in module_register — DECLARE_MODULE() in current module.h takes a moduledata_t variable, and the shipped kernel/module pair predates the toolchain state; the guest kernel was rebuilt from the current tree (nativekernel) before verification. Stock /boot/kernel modules load fine before and after.

Reachability of the driver-level trigger (source-traced)

• ifconfig tx0 down / mtu change / media change: root on a machine with the NIC; the tick fires 1/s and parks on the serializer whenever it is contended, so repeated down/up churn under load hits the window.
• epic_ifwatchdog: fires by itself (if_timer=8 set in epic_ifstart, if_tx.c:540) on TX stall, then calls epic_stop+epic_init under ifnet_tryserialize_all — no root needed, purely hardware timing; also wedges netisr0.
• epic_intr PCI-fatal: hardware-timing.

Fix (fix.diff, authored, not applied)

Convert the tick to a lock-aided callout (callout_init_lk with ifp->if_serializer after ether_ifattach) so the callback no longer serialize_enters (the callout core provides the interlock and its stop path breaks the AB-BA via LK_CANCEL), or alternatively stop the callout outside the serializer in every epic_stop() caller. fix.diff implements the former.

Bottom line: REPRODUCED — permanent kernel deadlock (DoS); the in-driver trigger is hardware-gated (tx(4) NIC) but the mechanism is proven on the guest kernel with the driver's exact discipline, matching the verified DF-3715 family.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.