# DF-4303 — bfe_stop() can deadlock the callout subsystem against bfe_tick (AB-BA), wedging the interface and a softclock CPU

**File:** `sys/dev/netif/bfe/if_bfe.c` (pass 2; known DF-1509 not touched)
**Lines:** 1486-1508 (`bfe_tick`: BLOCKING `lwkt_serialize_enter` at :1494, hz self re-arm INSIDE the serializer at :1497), 1516-1530 (`bfe_stop`: sync `callout_stop()` at :1522 under the same serializer), 323 (`callout_init`, non-MP), 1386 (arming), plus every `bfe_stop` caller: `bfe_ioctl` SIOCSIFFLAGS :1449-1450, `bfe_init` :1365 (itself called from `bfe_watchdog` :1480 and the `bfe_intr` error path :1186), `bfe_detach` :441, `bfe_shutdown` :475.
**Kernel refs:** `sys/kern/kern_timeout.c:1091` (`callout_stop` sync), `:859-930` (`_callout_cancel_or_stop`, `ssleep(..., "costp", 0)` at :920), `sys/kern/lwkt_serialize.c:105` (`lwkt_serialize_enter` → adaptive spin then sleep).

## The bug

`bfe_tick()` is a NON-MP callout (`callout_init`, if_bfe.c:323) armed at
`hz` (if_bfe.c:1386) whose callback **blocks** acquiring the driver
serializer (:1494) and re-arms itself **inside** it (:1497). `bfe_stop()`
calls `callout_stop(&sc->bfe_stat_timer)` (:1522) **while holding that
serializer**.

DragonFly's `callout_stop()` synchronously *waits for an in-progress
callback* (`_callout_cancel_or_stop`, sync=1 → `ssleep("costp")` until the
callback completes). If `bfe_tick` was dequeued (INPROG) and is blocked in
`lwkt_serialize_enter()` on the serializer held by the `bfe_stop` caller,
the two wait on each other forever:

```
softclock thread:  bfe_tick -> lwkt_serialize_enter(ser)   ... blocked (sleeping)
bfe_stop caller:   holds ser -> callout_stop() -> "costp"  ... blocked (sleeping)
```

Both threads wedge permanently; the ifnet serializer is never released
(every later driver entry point on bfe0 blocks), and the wedged softclock
thread stops delivering **all** callouts for its CPU (system-wide timer
starvation on that core). Identical protocol to the PROVEN DF-3715 family
(em), with bfe's own flavor (non-MP callout, hz re-arm inside).

**Reachable without root:** the watchdog path — `bfe_start` sets
`if_timer = 5` (:1351) on TX; `if_slowtimo_dispatch`
(sys/net/if.c:1900-1912) fires `bfe_watchdog` under
`ifnet_tryserialize_all` → `bfe_init` (:1480) → `bfe_stop` (:1365), and the
`bfe_intr` ISTAT_ERRORS path (:1186) does the same from interrupt context.
An unprivileged local user sending traffic onto a stalled bfe link sets
this up; `ifconfig bfe0 down` (root/devd) is another entry.

## Proven on the guest kernel (deterministic bfe-protocol replica)

`bfe_tick_replica.c` reproduces the exact bfe protocol (non-MP
`callout_init` + 1-tick first fire + **hz re-arm inside** the serializer +
**blocking** enter) and the `bfe_stop` shape (hold serializer →
`callout_stop`). The AB-BA test wedges exactly as predicted:

```
bferplica: loaded (hz=100)
bferplica: armed (tick: 1 tick then hz; fixed: 1 tick)
bferplica[fixed]: holding serializer, calling callout_stop()
bferplica[fixed]: callout_stop RETURNED (no deadlock), pending=0
bferplica[blk]: bfe_tick handler BLOCKED on our serializer (fired=16 ran=15) - entering callout_stop()
bferplica[blk]: IF THIS IS THE LAST bferplica LINE, AB-BA DEADLOCK IS PROVEN (bfe_tick vs bfe_stop)
  (no further output; "callout_stop returned (NO deadlock)" never printed)

ps -o pid,stat,wchan,command -ax | grep stop_blk
  1048 D2     costp    sysctl kern.bferplica.stop_blk=1
```

`costp` is the wait-channel inside `kern_timeout.c:920` — direct proof the
`bfe_stop`-shaped caller is stuck in `callout_stop()`'s sync wait while the
`bfe_tick`-shaped handler is stuck on its serializer. The ssh session used
to launch the background test itself stalled (callout starvation on the
affected CPU) — the guest had to be reset. **Positive control / fix
validation:** the try-enter ("fixed") shape is fully stopped by
`callout_stop()` under a held serializer (`pending=0`, returns) — the
family fix works on this kernel.

## Build/run (guest, DragonFly 6.5-DEVELOPMENT, /usr/src present)

```
cd /root/kmod && make            # build.log (BUILD_RC=0, -Werror clean)
kldload ./bfe_tick_replica.ko
sysctl kern.bferplica.arm=1
sysctl kern.bferplica.stop_fixed=1   # control: returns, pending=0
(sysctl kern.bferplica.stop_blk=1 &) # NEVER returns; ps shows D2 costp
# guest is wedged afterwards - vm.sh reset with-src
```