DF-4046 — stge_stop() deadlocks against stge_tick (synchronous callout_stop under the driver serializer)

• File: sys/dev/netif/stge/if_stge.c (stge(4))
• Severity: Medium · Confidence: certain · stge-specific runtime hardware-gated; mechanism PROVEN on the guest kernel via protocol replica
• Family: DF-3715 (em, proven) / DF-4025 (re, proven) — callout AB-BA

Build

cd /root/stgereplica && make      # bsd.kmod.mk module

Run

kldload /root/stgereplica/stgereplica.ko
sysctl kern.stgereplica.arm=1     # arm the stge_tick replica (1-tick period)
sleep 2
(sysctl kern.stgereplica.stop=1 >/tmp/stop4046.out 2>&1 &)   # NEVER returns
sleep 6
dmesg | grep stgereplica | tail -4
ps -o pid,stat,wchan,command -ax | grep "stop=1" | grep -v grep

Expected result (observed)

stgereplica: stge_tick parked on our serializer (fired=202 ran=201) - entering callout_stop()
stgereplica: IF THIS IS THE LAST stgereplica LINE, AB-BA DEADLOCK IS PROVEN
<<< console ends here — callout_stop() never returns >>>
...
43529 D0     costp    sysctl kern.stgereplica.stop=1   <- kern_timeout.c:920
    -1 B2     slize    (softclock 2)                   <- tick parked in
                                                            lwkt_serialize_enter

The observing shell itself never finishes its sleep (callout starvation); the guest afterwards cannot even shut down cleanly and must be force-reset (vm.sh revert with-src && vm.sh up).

On real stge(4) hardware

ifconfig stgeX down, SIOCSIFMTU/SIOCSIFCAP reinit, Tx-underrun force-init from the interrupt path (if_stge.c:1438-1440), detach, suspend, or reboot — each holds the serializer through stge_stop() — racing the 1 Hz stge_tick on another CPU wedges the kernel permanently: two threads stuck (costp × slize), the interface serializer never released, and the affected CPU's softclock stops delivering ALL callouts.

Artifacts

stgereplica.c, Makefile, build.sh, run.sh, build.log, run.log, env.txt, fix.diff, VERDICT.md, manifest.json, verdict.json.

DF-4046 VERDICT — stge_stop vs stge_tick callout AB-BA (mechanism PROVEN on guest; stge runtime hardware-gated)

Reproduced?

Mechanism: PROVEN on the guest kernel by a deterministic replica (stgereplica.ko) of if_stge.c's exact callout/serializer protocol — including the details that distinguish stge from the re(4) instance (DF-4025): the callout is initialized with plain non-MP callout_init (if_stge.c:587, dispatched by softclock under the MPLOCK, sys/kern/kern_timeout.c:575-590) and stge_tick re-arms itself unconditionally while holding the serializer (if_stge.c:1775 — no IFF_RUNNING gate at all, unlike re_tick_serialized).

stge-specific runtime: hardware-gated (guest has no Sundance/Tamarack NIC; env.txt), so per family convention the persisted status is inconclusive, matching DF-4025.

Observed on the guest (run.log + serial console)

1. Module built: BUILD_RC=0 (build.log).
2. Armed; tick fired 202 times, ran 201 (re-arm loop working, period 1 tick vs the real driver's hz).
3. stge_stop-shaped stop under the serializer: the 202nd firing parked in lwkt_serialize_enter; dmesg ends at IF THIS IS THE LAST stgereplica LINE, AB-BA DEADLOCK IS PROVEN — callout_stop() never returned.
4. Cross-CPU AB-BA visible in one ps: stopper pid 43529 D0 wchan costp (ssleep(c, &c->spin, 0, "costp", 0), sys/kern/kern_timeout.c:920, reached from callout_stop → _callout_cancel_or_stop(..., sync=1) at kern_timeout.c:1091-1093) vs (softclock 2) blocked at wchan slize (lwkt_serialize_enter) — each holds what the other needs, forever.
5. The observing shell never completed its sleep (timer starvation), and the wedged guest could not be shut down cleanly — vm.sh reset reported "guest still up after shutdown" and the guest had to be force-killed and reverted to the with-src snapshot (restored, healthy, guest_dirty=0 post-reset).

Root cause (path:line)

• if_stge.c:587 — callout_init(&sc->sc_tick_ch) (non-MP variant).
• if_stge.c:1752-1778 — stge_tick: lwkt_serialize_enter at :1758 (BLOCKING acquire), work, then unconditional callout_reset(..., hz, stge_tick, sc) at :1775 under the serializer, exit at :1777.
• if_stge.c:2098-2111 — stge_stop: ASSERT_SERIALIZED then callout_stop(&sc->sc_tick_ch) at :2111 as the FIRST teardown action, while holding the serializer; IFF_RUNNING is cleared only at the very end (:2166).
• Every stge_stop caller holds the serializer: stge_detach :820-825, stge_shutdown :1049-1051, stge_suspend :1060-1063, stge_ioctl SIOCSIFFLAGS :1280, stge_init :1900/:1907 — and stge_init is reachable from stge_ioctl SIOCSIFMTU :1265, the watchdog :1238, and the interrupt path's force_init (:1438-1440, host-error / Tx-underrun — remotely nudgeable by link events).
• DragonFly callout_stop() is synchronous (kern_timeout.c:1091-1093 → _callout_cancel_or_stop(..., sync=1); ssleep "costp" at :920): it waits for an INPROG callback to finish. A tick dequeued by a softclock thread and parked in lwkt_serialize_enter therefore deadlocks against the stopper: both threads wedge permanently, the interface serializer is never released (all later ioctls/if_start on stgeX block), and the affected CPU's softclock stops delivering ALL callouts. The unconditional re-arm (:1775) means even a late-running tick re-arms itself after any non-synchronous stop attempt.

Threat model

Local DoS via permanent kernel deadlock (two threads + CPU-wide callout starvation). Triggers are ordinary operator actions (ifconfig down, MTU/capability changes, detach, suspend, reboot) racing the 1 Hz tick — same privilege framing as DF-3715(em)/DF-4025(re), which were filed Medium. No memory-corruption primitive.

Fix validation

fix.diff makes stge_tick non-blocking: lwkt_serialize_try() with a 1-tick callout_reset retry when the serializer is busy — the same shape proposed and verified for em_timer (DF-3715) and re_tick (DF-4025). With a never-blocking tick, stge_stop's callout_stop can always complete. git apply --check passes against the read-only tree. A runtime fix-validation would need stge(4) hardware to race stge_stop against a real stge_tick; the replica's wedged protocol is exactly what the fix removes — fix_status: not_testable.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.