DF-4303 — bfe_stop() can deadlock the callout subsystem against bfe_tick (AB-BA), wedging the interface and a softclock CPU

File: sys/dev/netif/bfe/if_bfe.c (pass 2; known DF-1509 not touched) Lines: 1486-1508 (bfe_tick: BLOCKING lwkt_serialize_enter at :1494, hz self re-arm INSIDE the serializer at :1497), 1516-1530 (bfe_stop: sync callout_stop() at :1522 under the same serializer), 323 (callout_init, non-MP), 1386 (arming), plus every bfe_stop caller: bfe_ioctl SIOCSIFFLAGS :1449-1450, bfe_init :1365 (itself called from bfe_watchdog :1480 and the bfe_intr error path :1186), bfe_detach :441, bfe_shutdown :475. Kernel refs: sys/kern/kern_timeout.c:1091 (callout_stop sync), :859-930 (_callout_cancel_or_stop, ssleep(..., "costp", 0) at :920), sys/kern/lwkt_serialize.c:105 (lwkt_serialize_enter → adaptive spin then sleep).

The bug

bfe_tick() is a NON-MP callout (callout_init, if_bfe.c:323) armed at hz (if_bfe.c:1386) whose callback blocks acquiring the driver serializer (:1494) and re-arms itself inside it (:1497). bfe_stop() calls callout_stop(&sc->bfe_stat_timer) (:1522) while holding that serializer.

DragonFly's callout_stop() synchronously waits for an in-progress callback (_callout_cancel_or_stop, sync=1 → ssleep("costp") until the callback completes). If bfe_tick was dequeued (INPROG) and is blocked in lwkt_serialize_enter() on the serializer held by the bfe_stop caller, the two wait on each other forever:

softclock thread:  bfe_tick -> lwkt_serialize_enter(ser)   ... blocked (sleeping)
bfe_stop caller:   holds ser -> callout_stop() -> "costp"  ... blocked (sleeping)

Both threads wedge permanently; the ifnet serializer is never released (every later driver entry point on bfe0 blocks), and the wedged softclock thread stops delivering all callouts for its CPU (system-wide timer starvation on that core). Identical protocol to the PROVEN DF-3715 family (em), with bfe's own flavor (non-MP callout, hz re-arm inside).

Reachable without root: the watchdog path — bfe_start sets if_timer = 5 (:1351) on TX; if_slowtimo_dispatch (sys/net/if.c:1900-1912) fires bfe_watchdog under ifnet_tryserialize_all → bfe_init (:1480) → bfe_stop (:1365), and the bfe_intr ISTAT_ERRORS path (:1186) does the same from interrupt context. An unprivileged local user sending traffic onto a stalled bfe link sets this up; ifconfig bfe0 down (root/devd) is another entry.

Proven on the guest kernel (deterministic bfe-protocol replica)

bfe_tick_replica.c reproduces the exact bfe protocol (non-MP callout_init + 1-tick first fire + hz re-arm inside the serializer + blocking enter) and the bfe_stop shape (hold serializer → callout_stop). The AB-BA test wedges exactly as predicted:

bferplica: loaded (hz=100)
bferplica: armed (tick: 1 tick then hz; fixed: 1 tick)
bferplica[fixed]: holding serializer, calling callout_stop()
bferplica[fixed]: callout_stop RETURNED (no deadlock), pending=0
bferplica[blk]: bfe_tick handler BLOCKED on our serializer (fired=16 ran=15) - entering callout_stop()
bferplica[blk]: IF THIS IS THE LAST bferplica LINE, AB-BA DEADLOCK IS PROVEN (bfe_tick vs bfe_stop)
  (no further output; "callout_stop returned (NO deadlock)" never printed)

ps -o pid,stat,wchan,command -ax | grep stop_blk
  1048 D2     costp    sysctl kern.bferplica.stop_blk=1

costp is the wait-channel inside kern_timeout.c:920 — direct proof the bfe_stop-shaped caller is stuck in callout_stop()'s sync wait while the bfe_tick-shaped handler is stuck on its serializer. The ssh session used to launch the background test itself stalled (callout starvation on the affected CPU) — the guest had to be reset. Positive control / fix validation: the try-enter ("fixed") shape is fully stopped by callout_stop() under a held serializer (pending=0, returns) — the family fix works on this kernel.

Build/run (guest, DragonFly 6.5-DEVELOPMENT, /usr/src present)

cd /root/kmod && make            # build.log (BUILD_RC=0, -Werror clean)
kldload ./bfe_tick_replica.ko
sysctl kern.bferplica.arm=1
sysctl kern.bferplica.stop_fixed=1   # control: returns, pending=0
(sysctl kern.bferplica.stop_blk=1 &) # NEVER returns; ps shows D2 costp
# guest is wedged afterwards - vm.sh reset with-src

DF-4303 VERDICT — bfe_tick / bfe_stop callout AB-BA deadlock (DF-3715 family)

Bottom line

Mechanism PROVEN on the guest kernel by a deterministic replica of if_bfe.c's exact callout/serializer protocol; bfe-specific runtime is hardware-gated (no Broadcom BCM4401 exists on this guest — QEMU has no such device model — and none can be added), so per family convention (DF-3715) the overall verdict is inconclusive with the defect certain from source and the mechanism certain from execution.

Source trace (sys/dev/netif/bfe/if_bfe.c)

1. callout_init(&sc->bfe_stat_timer) — :323, non-MP flavor.
2. Armed callout_reset(&sc->bfe_stat_timer, hz, bfe_tick, sc) — :1386 (from bfe_init).
3. bfe_tick — :1486-1508: - mii = device_get_softc(sc->bfe_miibus) :1492 (outside serializer) - lwkt_serialize_enter(ifp->if_serializer) :1494 — BLOCKING (sys/kern/lwkt_serialize.c:105 → adaptive spin up to SLZ_ADAPTIVE_SPINMAX then lwkt_serialize_sleep) - callout_reset(..., hz, bfe_tick, sc) :1497 — self re-arm INSIDE - lwkt_serialize_exit :1508.
4. bfe_stop — :1516-1530: ASSERT_SERIALIZED :1520 → callout_stop :1522 (the sync variant, sys/kern/kern_timeout.c:1091 → _callout_cancel_or_stop(cc, CALLOUT_STOP, 1) :859 — waits ssleep(c, &c->spin, 0, "costp", 0) :920 until the INPROG callback completes).
5. bfe_stop callers holding the serializer: bfe_ioctl SIOCSIFFLAGS :1449-1450, bfe_init :1365 ← bfe_watchdog :1480 ← if_slowtimo_dispatch (sys/net/if.c:1900-1912, ifnet_tryserialize_all), bfe_intr ISTAT_ERRORS :1186, bfe_detach :441, bfe_shutdown :475.

Interleaving: tick dequeued → parks sleeping on the serializer → stopper enters serializer → callout_stop parks sleeping on "costp" → neither can ever proceed. The ifnet serializer is held forever by a sleeping thread; the softclock thread of the callback's CPU is consumed forever (no further callouts on that CPU).

Guest execution (this run, evidence in run.log / build.log)

• make in /root/kmod: BUILD_RC=0, -Werror clean (build.log).
• kldload OK; kern.bferplica.arm=1.
• Control (fix shape): kern.bferplica.stop_fixed=1 → bferplica[fixed]: callout_stop RETURNED (no deadlock), pending=0 — try-enter handlers are safely stopped under a held serializer → the family fix (fix.diff) is validated at mechanism level on this kernel.
• AB-BA (bfe shape): kern.bferplica.stop_blk=1 in background → last dmesg lines: bferplica[blk]: bfe_tick handler BLOCKED on our serializer (fired=16 ran=15) - entering callout_stop() bferplica[blk]: IF THIS IS THE LAST bferplica LINE, AB-BA DEADLOCK IS PROVEN (bfe_tick vs bfe_stop) and nothing after. ps: 1048 D2 costp sysctl kern.bferplica.stop_blk=1 — D-state thread parked on the kern_timeout.c:920 wait channel. The launching ssh session stalled (timer starvation on the affected CPU); guest required vm.sh reset with-src (performed; guest restored clean).

Threat model

Local DoS. The watchdog entry (TX timeout) is drivable by an unprivileged local user pumping traffic at a bfe interface whose link is stalled (bfe_start sets if_timer=5 :1351; watchdog → bfe_init → bfe_stop). Every bfe_stop races the 1 Hz tick; the race window is the callback's park-on-serializer interval. Root paths (ifconfig down, detach, shutdown) hit the same race. On hit: interface permanently wedged + one CPU's callout pipeline dead → system-wide degradation (observed: ssh stall).

No memory corruption: availability only. No uid=0 route (not a corruption class).

Why not false-positive / alternatives

• Not the "racing re-arm survives callout_stop" family: the DF-3715 control (and my stop_fixed control here, pending=0) shows this kern_timeout.c unloads a racing re-arm after STOP — the defect is purely the AB-BA.
• bfe_stop cannot simply switch to callout_stop_async: bfe_tick re-arms itself unconditionally (:1497), so an async stop leaves the timer cycling on a stopped/detached device. The complete fix is the try-enter handler (fix.diff), matching the DF-3715 fix merged for em(4).

Fix validation

fix.diff (try-enter + unconditional outside re-arm) — the fixed shape is exactly the stop_fixed positive control, which returns cleanly with pending=0 on this guest kernel. The driver-level diff cannot be runtime tested without BCM4401 hardware → fix_status not_testable (driver), mechanism validated (control).

Artifacts

• bfe_tick_replica.c + Makefile — the bfe-protocol replica module.
• build.log — full module build (+ harness build/run) output, BUILD_RC=0.
• run.log — load/arm/control/AB-BA session: dmesg lines + ps capture (1048 D2 costp).
• env.txt — guest uname, NIC inventory (vtnet0/lo0 only), no Broadcom PCI, md5(guest if_bfe.c) == md5(audited tree).
• fix.diff — bfe_tick try-enter fix (patch -p1 --dry-run clean).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.