DF-3738 — emx(4) callout_stop(&sc->timer) AB-BA deadlock (DF-3715 em(4) twin)

What

emx_timer() is an MP callout whose callback blocks on the driver serializer; emx_stop() (and two other paths) call callout_stop() on it while holding that same serializer. DragonFly's callout_stop() is synchronous — it sleeps ("costp", sys/kern/kern_timeout.c:920) until the in-progress callback finishes — and lwkt_serialize_enter() sleeps ("slize", sys/kern/lwkt_serialize.c:234) until the serializer is free. Callback parked on the serializer + stopper parked in callout_stop() holding the serializer = permanent AB-BA deadlock: the stopping thread AND the softclock thread of that CPU wedge forever.

Driver sites: - emx_timer() — sys/dev/netif/emx/if_emx.c:1908-1931 (lwkt_serialize_enter(&sc->main_serialize) at :1914, self re-arm at :1928 inside the serializer). - callout_stop(&sc->timer) under ASSERT_IFNET_SERIALIZED_ALL: - emx_stop() — :2030 (callers: SIOCSIFFLAGS :1227, emx_init :1354, emx_detach :989, emx_suspend :1059 — every ifconfig down/up, MTU/media/cap change, detach, kldunload, suspend, watchdog reinit); - emx_intr_body() LSC path — :1534 (ISR holds main_serialize; then emx_serialize_skipmain adds tx+rx serializers); - emx_npoll_status() — :4196 (polling status, main serializer).

The try-based emx_txgc_timer() (:426-449) is not affected (it never blocks); this is exactly the em(4) DF-3715 structure.

Build / run (guest)

# tarball Makefile + emx_callout_replica.c into /root/emxkmod
cd /root/emxkmod && make
kldload /root/emxkmod/emx_callout_replica.ko
sysctl kern.emxreplica.arm=1; sleep 2
sysctl kern.emxreplica.stop_gc=1            # negative control: returns
(sysctl kern.emxreplica.stop_blk=1 &)       # AB-BA: NEVER returns
sleep 6; dmesg | grep emxreplica | tail
ps -o pid,stat,wchan,command -ax | grep stop_blk    # D4 costp
# then reset the guest

Expected (and observed) output

Console ends with:

emxreplica[blk]: handler BLOCKED on our serializer (fired=14 ran=13) - entering callout_stop() [emx_stop shape]
emxreplica[blk]: IF THIS IS THE LAST emxreplica LINE, AB-BA DEADLOCK IS PROVEN

— and the "callout_stop returned" line NEVER appears. ps shows the stopper in D4 costp; ps -ax shows softclock 0 in B0 slize (parked inside the callback's lwkt_serialize_enter). Negative control (stop_gc): callout_stop returned, pending=0.

Fixed-shape validation (emx_callout_replica_fixed.c, replicating the fix.diff protocol): sysctl kern.emxreplica.stop_fixed=1 → callout_stop returned, pending=0, STOP_FIXED_RC=0, clean kldunload.

DF-3738 — VERDICT

Status: inconclusive per family convention (the emx device path is hardware-gated on this guest), but the deadlock MECHANISM is PROVEN live on the guest kernel via a deterministic protocol replica — same standard as the em(4) twin DF-3715.

Mechanism proof (guest, 2026-09-15, full transcript in run.log)

Kernel module emx_callout_replica.ko replicates if_emx.c's exact callout/serializer protocol with two callouts:

• blk_timer == emx_timer shape (if_emx.c:1908-1931): blocking lwkt_serialize_enter (:1914) + self re-arm INSIDE the serializer (:1928, before exit :1930);
• gc_timer == emx_txgc_timer shape (:426-449): non-blocking lwkt_serialize_try (:436) + unconditional done: re-arm (:448).

Observed on the guest (serial console + second session): 1. Negative control (sysctl kern.emxreplica.stop_gc=1, emx_stop shape against the try-based handler): callout_stop returned, pending=0 — non-blocking callbacks are torn down fine. 2. AB-BA (sysctl kern.emxreplica.stop_blk=1 in background): emxreplica[blk]: handler BLOCKED on our serializer (fired=14 ran=13) - entering callout_stop() [emx_stop shape] — then silence forever; the writing ssh session hung and was killed host-side; ps: 1105 D4 costp sysctl kern.emxreplica.stop_blk=1 (the synchronous sleep INSIDE callout_stop(), sys/kern/kern_timeout.c:920); ps -ax: (softclock 0) B0 slize — CPU0's softclock thread parked in lwkt_serialize_enter() (sys/kern/lwkt_serialize.c:234), i.e. INSIDE the emx_timer-shaped callback. Both sides of the AB-BA captured. 3. Guest recovered only via vm.sh reset with-src.

Why the real driver deadlocks (path:line)

• callout_stop() is synchronous: sys/kern/kern_timeout.c:1091 → _callout_cancel_or_stop(..., sync=1) (:859), ssleep(c, &c->spin, 0, "costp", 0) at :920 waits for the INPROG callback to finish.
• lwkt_serialize_enter() blocks: sys/kern/lwkt_serialize.c:234 (tsleep(s, PINTERLOCKED, "slize", 0)).
• emx_timer callback blocks on main_serialize (if_emx.c:1914); emx_stop()/emx_intr_body()/emx_npoll_status() call callout_stop(&sc->timer) (if_emx.c:2030/:1534/:4196) while holding it (ASSERT_IFNET_SERIALIZED_ALL at :2026; ISR + emx_serialize_skipmain at :1532; polling status serializer at :4236).
• Consequence: permanent wedge of the stopping thread (any ifconfig down/up, MTU/media/cap change, detach/kldunload, suspend, watchdog reinit — all reach emx_init→emx_stop or emx_stop directly) AND of one CPU's softclock thread (all future callouts on that CPU starve).

Race window: the 1 Hz timer must be INPROG-parked when the stop runs — small per call, but a down/up loop races it ~1/s per interface. Same judgment as DF-3715: Medium (system-level deadlock/DoS from ordinary privileged interface operations).

Fix validated at protocol level

fix.diff (git-apply clean against the read-only tree) converts emx_timer() to the non-blocking try-and-retry pattern already used by emx_txgc_timer(): lwkt_serialize_try(&sc->main_serialize); on failure re-arm in 1 tick and return; double-check IFF_RUNNING|IFF_UP both before the try (cheap racy read) and under the serializer.

Protocol validation on the guest (emx_callout_replica_fixed.c, the fixed-shape replica): sysctl kern.emxreplica.stop_fixed=1 → emxreplica[fx]: callout_stop returned, pending=0, emxreplica[fx]: RESULT pending=0 (fix effective), STOP_FIXED_RC=0, clean kldunload. Contrast with the unfixed shape under the identical stop: permanent D4/costp wedge. (fix_status therefore fixed at the protocol-replica level; a full kernel rebuild validation is not possible without emx hardware — noted in fix_verdict.)

Note on the gc timer: the try-based emx_txgc_timer re-arm (:448) cannot survive callout_stop post-processing on this kern_timeout.c (negative control, pending=0) — the em(4) pass-2 reached the same conclusion, so no separate post-stop re-arm finding is filed.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.