DF-4310 — ste(4) callout AB-BA (DF-3715 family) — evidence pack

Finding: ste_stop() / ste_intr() STATS_OFLOW path issue the synchronous callout_stop(&sc->ste_stat_timer) while holding ifp->if_serializer, while the 1 Hz non-MP ste_stats_update callback must first block in lwkt_serialize_enter() on that same serializer (holding the MP lock) — permanent AB-BA deadlock; secondary: the tick's in-body callout_reset overrides a racing STOP and resurrects the timer after detach (UAF).

Build (on the guest, /usr/src present)

# copy ste_callout_replica.c + Makefile to /root/kmod on the guest
cd /root/kmod && make           # -Werror clean, see build.log

Run (on the guest)

kldload /root/kmod/ste_callout_replica.ko
sysctl kern.sterplica.arm=1
sleep 2
sysctl kern.sterplica.stop_fixed=1      # positive control: returns
(sysctl kern.sterplica.stop_blk=1 &)    # NEVER returns -> wedge
sleep 10; dmesg | grep sterplica | tail -8
ps -o pid,stat,wchan,command -ax | grep stop_blk
# guest now wedged (D4/costp thread + softclock thread + MP lock) — reset:
#   vm.sh reset with-src

Expected decisive output (observed, run.log)

sterplica[fixed]: callout_stop RETURNED (no deadlock), pending=0
sterplica[blk]: ste_stats_update handler BLOCKED on our serializer (fired=14 ran=13) - entering callout_stop()
sterplica[blk]: IF THIS IS THE LAST sterplica LINE, AB-BA DEADLOCK IS PROVEN (ste_stats_update vs ste_stop)
1076 D4     costp    sysctl kern.sterplica.stop_blk=1

Contents

• ste_callout_replica.c / Makefile — protocol replica module
• build.sh / build.log — guest build (-Werror clean, BUILD_RC=0)
• run.sh / run.log — full session: presence probe, control, wedge, ps
• env.txt — guest uname, clean-reset confirmation
• VERDICT.md — full narrative with path:line trace
• fix.diff — try-enter + re-arm-outside fix (git-apply-able)
• manifest.json, verdict.json

DF-4310 — ste(4) callout AB-BA deadlock (DF-3715 family instance)

File: sys/dev/netif/ste/if_ste.c (Sundance ST201 / D-Link DFE-550TX)

What was tested and how

The guest has no ST201 hardware (pciconf -l → NO_SUNDANCE_PCI; QEMU cannot emulate the chip), so the driver-attached path cannot run. Per DF-3715 family convention, the callout/serializer protocol if_ste.c uses was replicated 1:1 in an out-of-tree kernel module (ste_callout_replica.c) and exercised on the guest kernel — the deadlock lives in the generic protocol, not in the NIC registers.

Replicated protocol elements (exact if_ste.c line numbers):

ste_stop() is reached with ifp->if_serializer held from ste_detach (:997-998), ste_init (:1130, reached from ste_ioctl SIOCSIFFLAGS :1322/:1326, ste_intr HOSTERR :617, ste_txeoc :723/:736, ste_watchdog :1494) and ste_shutdown (:1507). ste_watchdog itself runs serialized on netisr0 (sys/net/if.c if_slowtimo_dispatch → ifnet_tryserialize_all). ste_intr is INTR_MPSAFE with ifp->if_serializer (:974-976), so the STATS_OFLOW callout_stop (:608) executes in interrupt thread context under the serializer.

Result — MECHANISM PROVEN (see run.log)

1. Positive control (fix shape): the try-enter timer is fully unloaded by callout_stop() issued under a held serializer: sterplica[fixed]: callout_stop RETURNED (no deadlock), pending=0
2. ste shape (AB-BA): with the ste_stats_update-shaped handler parked on the serializer (fired=14 ran=13), callout_stop() NEVER RETURNED: sterplica[blk]: ste_stats_update handler BLOCKED on our serializer (fired=14 ran=13) - entering callout_stop() sterplica[blk]: IF THIS IS THE LAST sterplica LINE, AB-BA DEADLOCK IS PROVEN (ste_stats_update vs ste_stop) ps shows the caller permanently asleep in the synchronous-stop wait: 1076 D4 costp sysctl kern.sterplica.stop_blk=1 (costp = sys/kern/kern_timeout.c:920 ssleep(...,"costp",...) inside _callout_cancel_or_stop(sync=1) reached from callout_stop :1091.)
3. Corroboration: the wedged guest could no longer perform a clean shutdown -p now — ste_stats_update is a non-MP callout (callout_init at :914), so its callback runs holding the MP lock (sys/kern/kern_timeout.c:580-585) and parks on the serializer still holding it; every non-MPSAFE shutdown/system path then blocks. QEMU had to be force-killed and the disk reverted (vm.sh reset with-src).

Guest reset to the clean with-src snapshot afterwards (verified: dmesg | grep -c sterplica → 0, module not loaded).

Why the driver instance follows deterministically

• callout_stop() is the synchronous flavor in DragonFly (sys/kern/kern_timeout.c:1091 → _callout_cancel_or_stop(..., sync=1) → ssleep(c,&c->spin,0,"costp",0) :911-921): it waits for an in-progress callback to finish.
• ste_stats_update is non-MP (holds MP lock) and its first action (:790) is the blocking lwkt_serialize_enter(ifp->if_serializer).
• ste_stop/ste_intr call that callout_stop while holding the very serializer the tick needs → circular wait: stopper sleeps in costp holding the serializer; tick sleeps on the serializer holding the MP lock. Permanent: interface wedged, one softclock thread stops delivering all callouts for its CPU, netisr0 (watchdog path) and every MP-lock consumer stall.

Timing feasibility in the real driver: the 1 Hz tick is dequeued/INPROG on every boundary; ste_stop holds the serializer for ≥100 ms of busy-wait (ste_reset's DELAY(100000) at :1244/:1282) on every ifconfig ste0 down, ste_init, or watchdog/TX-error reinit, and the mii-poll body of ste_stats_update itself extends the INPROG window — collision is expected under ordinary admin activity or unprivileged TX traffic (ste_start sets if_timer=5 :1475 → watchdog → ste_init → ste_stop).

Secondary outcome (same root cause, source-level)

If the STOP lands while the tick is INPROG inside the body, the callback's own callout_reset (:811) is processed recursively and overrides the STOP (kern_timeout.c: RESET beats STOP; callout_stop's own comment documents re-arm-before-return semantics). ste_stop then returns with the timer armed, and at detach (ste_detach frees ste_ldata via contigfree :1012-1015, deletes the miibus child :1004-1005, releases the I/O resource :1010-1011) the resurrected tick dereferences the freed softc/miibus one second later — post-detach use-after-free/panic.

Fix

fix.diff (git-apply-able, dry-run clean): make ste_stats_update never block on the serializer (lwkt_serialize_try, skip the tick's work when contended) and re-arm OUTSIDE the serializer — exactly the shape validated by the positive control on this guest and the fix shape merged for em(4) in DF-3715.

Status classification

Mechanism proven on the guest kernel; the ste-specific runtime path is hardware-gated (no ST201 NIC on any QEMU backend). Per family convention (DF-3715/DF-4046/DF-4051/DF-4064/DF-4303): status inconclusive, reproduced=0 at the driver level, mechanism certainty certain.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.