DF-3715 — em_stop() can deadlock the callout subsystem against em_timer / em_82547_move_tail (AB-BA)

File: sys/dev/netif/em/if_em.c Lines: 2216-2239 (em_timer: blocking lwkt_serialize_enter at :2222, self re-arm at :2236), 2027-2036 (em_82547_move_tail: blocking enter at :2033, re-arm via :2016-2017), 2343-2350 (em_stop: callout_stop() under the same serializer), 1047-1065 (em_detach) Kernel refs: sys/kern/kern_timeout.c:859-932 (_callout_cancel_or_stop sync wait, ssleep(..., "costp", 0) at :923), :1091-1098 (callout_stop)

The bug

em_timer() is armed with callout_init_mp() (if_em.c:548) and its callback blocks acquiring the driver serializer (:2222). em_stop() — reached from every down/detach/MTU-change/reinit path (SIOCSIFFLAGS :1295, SIOCSIFMTU :1280, watchdog :1407, suspend :1112, detach :1049) — calls callout_stop(&adapter->timer) (:2343) while holding that serializer.

DragonFly's callout_stop() synchronously waits for an in-progress callback (_callout_cancel_or_stop, sync=1 → ssleep("costp") until the callback completes). If em_timer was dequeued (INPROG) and is blocked in lwkt_serialize_enter() on the serializer held by the em_stop caller, the two wait on each other forever:

softclock thread:  em_timer -> lwkt_serialize_enter(ser)  ... blocked
em_stop caller:    holds ser -> callout_stop() -> ssleep("costp") ... blocked

Both threads wedge permanently; the serializer is never released (every later driver entry point on that interface blocks), and the wedged softclock thread stops delivering all callouts for its CPU (system-wide timer starvation). The 82547 tx_fifo_timer has the same shape (em_82547_move_tail :2033). (em_txgc_timer uses non-blocking lwkt_serialize_try — immune.)

Proven on the guest kernel (deterministic kernel-module replica)

callout_replica.c reproduces the exact protocol (MP callout + hand-rolled serializer): kern.emreplica.stop_blk=1 holds the serializer, waits until the em_timer-shaped callback is parked inside lwkt_serialize_enter, then calls callout_stop() — which never returns:

replica[blk]: handler BLOCKED on our serializer (fired=91 ran=90) - entering callout_stop()
replica[blk]: IF THIS IS THE LAST replica LINE, AB-BA DEADLOCK IS PROVEN
  (no further output; "callout_stop returned (NO deadlock)" never printed)

ps -o pid,stat,wchan,command -ax | grep stop_blk
  1051 D3     costp    sysctl kern.emreplica.stop_blk=1

costp is the wait-channel inside kern_timeout.c:923 — direct proof the caller is stuck in callout_stop()'s sync wait. Negative control on the same module: the try-based (em_txgc_timer-shaped) handler is fully unloaded by callout_stop() (pending=0 after return) — so the try-based timers are fine, the blocking-enter handlers are the defect.

Build/run (guest, DragonFly 6.5-DEVELOPMENT, /usr/src present)

cd /root/kmod && make            # build.log
kldload ./callout_replica.ko     # loads fine on a fresh boot
sysctl kern.emreplica.arm=1
sysctl kern.emreplica.stop_gc=1  # negative control (returns)
(sysctl kern.emreplica.stop_blk=1 &)   # DEADLOCK - second session: ps wchan

No em hardware required — the mechanism is the generic callout/serializer protocol if_em.c uses; em-specific reachability is the source-proven race window (timer dequeued in the µs before em_stop's serializer acquire wins over the callback's acquire).

Fix

fix.diff: convert em_timer() and em_82547_move_tail() to non-blocking lwkt_serialize_try() with a 1-tick retry on contention (the exact pattern em_txgc_timer already uses) — a callback that never blocks can always complete, so callout_stop()'s sync wait always terminates.

DF-3715 — VERDICT

Status: mechanism PROVEN on the guest kernel via deterministic replica module (same callout/serializer protocol as if_em.c); em-specific runtime hardware-gated → inconclusive per family convention, reproduced=0, impact=none-on-this-guest. The class of impact, when it fires on em hardware, is a permanent kernel deadlock (DoS).

Root cause (path:line)

1. sys/dev/netif/em/if_em.c:548-550 — all three callouts are MP-safe (callout_init_mp), i.e. handlers run on a softclock thread and manage locking by hand.
2. sys/dev/netif/em/if_em.c:2222 — em_timer() does a blocking lwkt_serialize_enter(ifp->if_serializer); :2033 — em_82547_move_tail() (tx_fifo_timer callback) likewise.
3. sys/dev/netif/em/if_em.c:2343 and :2344 — em_stop() calls callout_stop(&adapter->timer) / callout_stop(&adapter->tx_fifo_timer) while holding the same serializer (em_stop requires ASSERT_SERIALIZED, :2339). Callers of em_stop: em_init (:1422 — every reinit: SIOCSIFFLAGS :1292-1295, SIOCSIFMTU :1279-1280, media change :1795-1796, watchdog :1407, npoll :1581-1582), em_detach (:1049), em_suspend (:1112).
4. sys/kern/kern_timeout.c:859-932 — _callout_cancel_or_stop(..., sync=1) (which callout_stop at :1091 uses): when the callout is INPROG, the caller ssleep(c, &c->spin, 0, "costp", 0) until the callback completes (:923).
5. ⇒ If em_timer is INPROG and parked in lwkt_serialize_enter() on the serializer held by the em_stop caller: the callback waits for the serializer, the stopper waits (costp) for the callback — permanent AB-BA deadlock. Collateral: the interface serializer is held forever (all later driver paths on that interface block), and the softclock thread of the CPU that armed the timer stops firing every callout on that CPU.

Race window: em_timer must be dequeued-and-not-yet-inside within the µs around em_stop's serializer acquire — once per second of runtime per em_stop invocation; hit rate is low but the trigger (ifconfig down/up, MTU change on a running interface, detach, suspend/resume, watchdog reinit) is repeatable, making eventual occurrence plausible on long-lived boxes.

Why this is NOT the DF-3669/DF-3681 finding

The ix/igb pass-2 findings asserted a re-arm-after-stop UAF on callout_stop grounds ("does not interlock against an in-progress callback"). On this kern_timeout.c, callout_stop()'s STOP flag is processed after the callback returns and unloads a racing re-arm (_callout_update_spinlocked, RESET-then-STOP ordering) — verified live by the replica's negative control (stop_gc: pending=0 after stop). For em the actual defect is therefore not a post-stop re-arm, but the sync-wait deadlock above. (em_detach also frees everything only after em_stop returns, which under this analysis it always does — no stop-side UAF.)

Verification performed (guest, DragonFly 6.5-DEVELOPMENT x86_64)

• callout_replica.c (this pack) — exact protocol replica; see run.log:
• negative control (try-based gc shape): stop fully unloaded → no bug for em_txgc_timer;
• positive test (blocking em_timer shape): callout_stop() never returned; caller in state D3 wchan costp (kern_timeout.c:923) — deadlock proven deterministically;
• guest wedged (sysctl thread + softclock thread), evidence captured from a second session, then vm.sh reset with-src.
• em hardware absent on guest (ifconfig -l → vtnet0 lo0; no 8086 NIC) — the if_em.c instance of the race itself cannot be fired here.

Fix

fix.diff (applies clean to read-only sys/): make em_timer() and em_82547_move_tail() non-blocking — lwkt_serialize_try() with a 1-tick callout_reset retry on contention, exactly the em_txgc_timer pattern (:494-506). A never-blocking callback always completes, so callout_stop's sync wait always terminates. Not hardware-testable on this guest (fix_status: not_testable).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.