DF-3850 — dc(4) AB-BA callout deadlock (dc_stop vs dc_tick)

Severity: Medium · confidence: certain (mechanism proven on guest) · hardware-gated at driver level (inconclusive per family convention)

What this is

sys/dev/netif/dc/if_dc.c — dc_stop() can deadlock the callout subsystem against dc_tick() (AB-BA): callout_stop() synchronously waits for a callback that is blocked on the very serializer the stopper holds. DF-3715/3738/3760/3776/3806 family (em, emx, alc, jme, msk); dc(4) instance, aggravated by dc's non-MPSAFE callout (parked softclock thread holds the Giant MP lock).

Reproduce (on the audit guest, or any DragonFly 6.x with /usr/src)

./build.sh          # build /root/kmod/dc_callout_replica.ko on the guest
./run.sh            # arm + fire the dc_stop shape in background;
                    # observe D1/costp stopper and B0/slize softclock 0

Success criterion (all observed on 2026-09-16, first attempt):

• serial console: dcreplica[stop]: tick INPROG and PARKED on the serializer (fired=21 ran=20) followed by IF THIS IS THE LAST dcreplica LINE callout_stop NEVER RETURNED and permanent silence from the replica;
• ps: stopper in state D1 wchan costp (kern_timeout.c:920), (softclock 0) in state B0 wchan slize (lwkt_serialize park);
• guest can no longer shut down cleanly (MP lock held) — requires vm.sh reset with-src.

Fix validation

dc_callout_replica_fixed.c + fix_validation.log: with the fix shape (lwkt_serialize_try + 1-tick retry), callout_stop() returns in 3.09 s, callout_pending()==0 — no deadlock, timer genuinely stopped.

Files

dc_callout_replica.c        deadlock replica module (dc protocol)
dc_callout_replica_fixed.c  fix-shape replica module
build.sh / run.sh           exact commands
build.log / run.log / fix_validation.log   full untrimmed logs
fix.diff                    git-apply-able fix vs read-only sys/ tree
VERDICT.md                  full narrative
verdict.json                machine verdict

DF-3850 — VERDICT

Status: mechanism PROVEN on the guest kernel via deterministic replica of if_dc.c's exact callout/serializer protocol (first attempt, no retries needed); dc-specific runtime hardware-gated → inconclusive per family convention. reproduced=0 (no dc NIC on the audit guest), impact on this guest=none. The class of impact, when it fires on dc hardware, is a permanent kernel deadlock (DoS): wedged interface, wedged stopper thread, softclock blackout on one CPU, and — because dc_stat_timer is a non-MPSAFE callout — the Giant MP lock held by the parked softclock thread, which wedged even the guest's clean shutdown.

Root cause (path:line)

1. sys/dev/netif/dc/if_dc.c:1847 — callout_init(&sc->dc_stat_timer): plain (non-MPSAFE) callout → the callback executes under the Giant MP lock (sys/kern/kern_timeout.c:583 get_mplock()).
2. sys/dev/netif/dc/if_dc.c:2725 — dc_tick() does a blocking lwkt_serialize_enter(ifp->if_serializer) (sys/kern/lwkt_serialize.c:105 → atomic_intr_cond_enter(…, lwkt_serialize_sleep) → tsleep, wchan slize); it self-re-arms at :2786-2789 and releases at :2791.
3. sys/dev/netif/dc/if_dc.c:3489 — dc_stop() calls callout_stop(&sc->dc_stat_timer) while holding the same serializer. Every dc_stop() caller holds it: dc_detach :2213, dc_intr :2949, dc_init :3196 (reached from SIOCSIFFLAGS-up via dc_ioctl :3432, watchdog dc_watchdog :3468, media/MTU reinit), dc_shutdown :3546, dc_suspend :3564.
4. sys/kern/kern_timeout.c:1091 → :859-932 — callout_stop() is synchronous: when the callout is INPROG (:567), the caller ssleep(c, &c->spin, 0, "costp", 0) (:920) until the callback completes.
5. ⇒ A dc_tick dequeued and parked in lwkt_serialize_enter() on the serializer held by the dc_stop caller deadlocks against it (AB-BA): both sleep forever; the interface serializer is never released; the affected CPU's softclock thread stops delivering all callouts; and since dc_stat_timer is non-MPSAFE, the parked softclock thread also holds the Giant MP lock, degrading the whole system (on the replica run the guest could no longer perform a clean shutdown).

Trigger on dc hardware: any dc_stop() racing a tick — ifconfig dcX down/up, MTU/media change on a running interface, watchdog reinit, detach/kldunload, suspend/resume, shutdown. The tick fires at 1 Hz (10 Hz with DC_21143_NWAY), and spends most of its body under the serializer (MII bit-banging with DELAY loops), so the window per stop is real; each interface stop is one lottery draw.

This is the dc(4) instance of the DF-3715/3738/3760/3776/3806 family (em, emx, alc, jme, msk). Aggravating delta vs those: dc's timer is a non-MPSAFE callout, so the parked callback additionally holds the Giant MP lock.

Verification performed (guest, DragonFly 6.5-DEVELOPMENT x86_64)

• dc_callout_replica.c (this pack) — kernel module replicating the dc protocol byte-for-byte in shape: plain callout_init, tick callback doing blocking lwkt_serialize_enter + self re-arm, stop sysctl emulating dc_stop (serializer held across callout_stop).
• Positive test (dc shape): first attempt reproduced — serial console: dcreplica[stop]: tick INPROG and PARKED on the serializer (fired=21 ran=20) then IF THIS IS THE LAST dcreplica LINE … and nothing more: callout_stop() never returned.
• Wedged-thread capture (second session): stopper 1530 D1 costp sysctl kern.dcreplica_stop=1 and (softclock 0) B0 slize — both halves of the cycle visible; softclock 1..5 still delivering.
• Collateral: the wedged guest could not be shut down cleanly (shutdown hung behind the parked state); qemu had to be SIGKILLed and the disk snapshot-reverted.
• Fix-shape validation — dc_callout_replica_fixed.c replicates the fixed protocol (lwkt_serialize_try + 1-tick retry, the exact shape of this pack's fix.diff): holding the serializer for 3 s produced 318 fires / 308 retries / 0 parks, callout_stop() returned in 3.09 s with callout_pending()==0 — deadlock eliminated, timer truly stopped. (fix_validation.log)
• dc hardware absent on this guest (ifconfig -l → vtnet0 lo0; pciconf -lv shows only QEMU i440fx + virtio devices), so the if_dc.c-specific race cannot be fired end-to-end here — hence inconclusive per family convention (identical to DF-3715/3738/3760/ 3776/3806).

Not a re-report

DF-3715/3738/3760/3776/3806 covered em(4), emx(4), alc(4), jme(4), msk(4). if_dc.c was never covered; its instance additionally demonstrates the non-MPSAFE-callout (Giant-MP-lock-holding) aggravation, which the prior family members (all callout_init_mp) do not have.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.