# DF-3715 — em_stop() can deadlock the callout subsystem against em_timer / em_82547_move_tail (AB-BA)

**File:** `sys/dev/netif/em/if_em.c`
**Lines:** 2216-2239 (`em_timer`: blocking `lwkt_serialize_enter` at :2222, self re-arm at :2236), 2027-2036 (`em_82547_move_tail`: blocking enter at :2033, re-arm via :2016-2017), 2343-2350 (`em_stop`: `callout_stop()` **under the same serializer**), 1047-1065 (`em_detach`)
**Kernel refs:** `sys/kern/kern_timeout.c:859-932` (`_callout_cancel_or_stop` sync wait, `ssleep(..., "costp", 0)` at :923), `:1091-1098` (`callout_stop`)

## The bug

`em_timer()` is armed with `callout_init_mp()` (if_em.c:548) and its callback
**blocks** acquiring the driver serializer (:2222). `em_stop()` — reached
from every down/detach/MTU-change/reinit path (SIOCSIFFLAGS :1295, SIOCSIFMTU
:1280, watchdog :1407, suspend :1112, detach :1049) — calls
`callout_stop(&adapter->timer)` (:2343) **while holding that serializer**.

DragonFly's `callout_stop()` synchronously *waits for an in-progress callback*
(`_callout_cancel_or_stop`, sync=1 → `ssleep("costp")` until the callback
completes). If `em_timer` was dequeued (INPROG) and is blocked in
`lwkt_serialize_enter()` on the serializer held by the `em_stop` caller, the
two wait on each other forever:

```
softclock thread:  em_timer -> lwkt_serialize_enter(ser)  ... blocked
em_stop caller:    holds ser -> callout_stop() -> ssleep("costp") ... blocked
```

Both threads wedge permanently; the serializer is never released (every later
driver entry point on that interface blocks), and the wedged softclock thread
stops delivering **all** callouts for its CPU (system-wide timer starvation).
The 82547 `tx_fifo_timer` has the same shape (`em_82547_move_tail` :2033).
(`em_txgc_timer` uses non-blocking `lwkt_serialize_try` — immune.)

## Proven on the guest kernel (deterministic kernel-module replica)

`callout_replica.c` reproduces the exact protocol (MP callout + hand-rolled
serializer): `kern.emreplica.stop_blk=1` holds the serializer, waits until
the em_timer-shaped callback is parked inside `lwkt_serialize_enter`, then
calls `callout_stop()` — which **never returns**:

```
replica[blk]: handler BLOCKED on our serializer (fired=91 ran=90) - entering callout_stop()
replica[blk]: IF THIS IS THE LAST replica LINE, AB-BA DEADLOCK IS PROVEN
  (no further output; "callout_stop returned (NO deadlock)" never printed)

ps -o pid,stat,wchan,command -ax | grep stop_blk
  1051 D3     costp    sysctl kern.emreplica.stop_blk=1
```

`costp` is the wait-channel inside `kern_timeout.c:923` — direct proof the
caller is stuck in `callout_stop()`'s sync wait. Negative control on the same
module: the try-based (`em_txgc_timer`-shaped) handler is fully unloaded by
`callout_stop()` (pending=0 after return) — so the try-based timers are fine,
the blocking-enter handlers are the defect.

## Build/run (guest, DragonFly 6.5-DEVELOPMENT, /usr/src present)

```
cd /root/kmod && make            # build.log
kldload ./callout_replica.ko     # loads fine on a fresh boot
sysctl kern.emreplica.arm=1
sysctl kern.emreplica.stop_gc=1  # negative control (returns)
(sysctl kern.emreplica.stop_blk=1 &)   # DEADLOCK - second session: ps wchan
```

No em hardware required — the mechanism is the generic callout/serializer
protocol if_em.c uses; em-specific reachability is the source-proven race
window (timer dequeued in the µs before `em_stop`'s serializer acquire wins
over the callback's acquire).

## Fix

`fix.diff`: convert `em_timer()` and `em_82547_move_tail()` to non-blocking
`lwkt_serialize_try()` with a 1-tick retry on contention (the exact pattern
`em_txgc_timer` already uses) — a callback that never blocks can always
complete, so `callout_stop()`'s sync wait always terminates.