β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-4260

vr_stop() invokes the synchronous callout_stop() while holding if_serializer: permanent kernel deadlock (CPU-local timer death) against vr_tick() blocked on that serializer

Summary

vr_tick begins with a blocking lwkt_serialize_enter and re-arms itself; vr_stop calls callout_stop and is always entered with the serializer held. On this kernel callout_stop is synchronous (contradicting its own doc block): when the callback is INPROG it sleeps in 'costp' WHILE THE CALLER STILL HOLDS the serializer the callback needs. Neither side can progress: the stopper sleeps forever holding the serializer, the softclock thread of one CPU wedges acquiring it, all timers on that CPU are dead for the life of the boot, and the triggering process is unkillable. DETERMINISTICALLY REPRODUCED on the 6-CPU guest with a pattern module (sysctl never returns, D3/costp, pinned sleep(3) never wakes on the wedged CPU, collateral oidlk blocks). The identical tick/stop shape exists in other netif drivers (rl, etc.) - the family precedent is DF-3715; this is the vr instance. stop_async + drain-outside fix (validated live, if_vr.ko compile-checked).

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-4260 Β· 11 files
FileTypeDescriptionSize
poc_deadlock.c β€” 4.6 KB view raw
pincpu.c β€” 788 B view raw
Makefile β€” 60 B ↓ download
build.sh β€” 174 B view raw
run.sh β€” 240 B view raw
build.log β€” 1.6 KB view raw
run.log β€” 959 B view raw
run.2.log β€” 874 B view raw
env.txt β€” 502 B view raw
fix.diff β€” 1.2 KB view raw
VERDICT.md β€” 5.9 KB ↓ raw
VERDICT.md
↓ download raw

VERDICT β€” DF-4260

Status: reproduced (deterministic). Impact: dos (permanent kernel deadlock). Guest: DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 2026, x86_64, hw.ncpu=6 (X86_64_GENERIC), fully updated audit snapshot.

What was verified

The vr(4) deadlock pattern was reproduced deterministically on the shipping guest kernel with a kld module (poc_deadlock.c) that replicates the exact code shape of if_vr.c β€” a callout callback that (1) re-arms itself and (2) lwkt_serialize_enter()s a serializer, plus an administrative path that holds that same serializer and calls the (synchronous) callout_stop():

  1. kldload poc_deadlock.ko
  2. sysctl kern.poc_deadlock=2 β€” fixed pattern (callout_stop_async() under the serializer + callout_drain() outside, exactly the remedy in fix.diff): returns cleanly, dmesg fixed pattern completed, state=2.
  3. sysctl kern.poc_deadlock=1 β€” bugged pattern (synchronous callout_stop() under the serializer, i.e. what vr_stop() at sys/dev/netif/vr/if_vr.c:1630 does on every ifconfig vrX down / kldunload / interrupt / watchdog path): hangs forever. - The sysctl process is stuck in state D3, wait channel costp β€” the ssleep(c, &c->spin, 0, "costp", 0) in _callout_cancel_or_stop() (sys/kern/kern_timeout.c:920). Unkillable (sleep has no PCATCH); timeout(1) itself blocks forever in wait() (pid 1704 pause). - The softclock thread of cpu4 (where the tick was INPROG) is wedged acquiring the serializer held by the sleeping stopper. Witness: a usched_set-pinned sleep 3 returns in 3s on cpu0,1,2,3,5 but never wakes on cpu4 (rc=124 only via timeout's SIGTERM, run.2.log) β€” cpu4's timer wheel is dead for the life of the boot. - Collateral: any later write to the same sysctl OID blocks forever on oidlk (pid 1735, D5) because the stuck writer holds the OID lock.

Why it happens (code trace)

  • vr_tick() starts with lwkt_serialize_enter(ifp->if_serializer) (sys/dev/netif/vr/if_vr.c:1165) and re-arms itself (if_vr.c:1178).
  • softclock marks the callout CALLOUT_INPROG before invoking the callback (sys/kern/kern_timeout.c:566-568), so a callback blocked on the serializer keeps INPROG set.
  • vr_stop() calls callout_stop(&sc->vr_stat_timer) (if_vr.c:1630) and is always called with ifp->if_serializer held: vr_detach (if_vr.c:822-826), vr_ioctl(SIOCSIFFLAGS) β€” ifioctl dispatches if_ioctl under ifnet_serialize_all() (sys/net/if.c:2168-2170) β€”, vr_intr (if_vr.c:1195), vr_watchdog (if_vr.c:1609), vr_init (if_vr.c:1408).
  • On this kernel callout_stop() is synchronous (_callout_cancel_or_stop(cc, CALLOUT_STOP, 1), sys/kern/kern_timeout.c:1091-1094): with INPROG set and the caller not being the softclock thread it ssleep("costp")s (kern_timeout.c:910-920) while the caller still holds the lwkt serializer the callback needs. lwkt serializers have no owner recursion/wakeup path for a sleeping holder (sys/kern/lwkt_serialize.c:105-111, adaptive spin then sleep) β†’ neither side can ever progress.
  • The documentation block directly above callout_stop() still claims it "does not block / does not interlock against a callout that is in-progress" (kern_timeout.c:1086-1093) β€” the code contradicts it; drivers were written against the documented (async) semantics. callout_stop_async() (kern_timeout.c:1060-1063) preserves the documented behavior.

SMP is required (the callback must be dequeued by another CPU's softclock while the stopper holds the serializer) β€” satisfied on any multi-core box.

vr(4)-specific reachability

  • ifconfig vrX down (root) β†’ vr_ioctl β†’ vr_stop under serializer β€” 1 Hz tick race window of ~Β΅s per attempt (statistical, ~1e-6..1e-4/s while hammering; the vr hardware itself could not be exercised β€” QEMU 7.2 has no via-rhine PCI device).
  • kldunload if_vr (root) β†’ vr_detach β€” same.
  • vr_intr β†’ vr_stop when IFF_UP was cleared (if_vr.c:1194-1197): the synchronous stop is then executed from the interrupt thread, sleeping in interrupt context while holding the serializer β€” the nastiest variant.
  • vr_watchdog (TX stall) β†’ vr_stop under serializer (if_vr.c:1609).
  • The identical tick-takes-serializer / stop-under-serializer shape exists in other DFly netif drivers (e.g. sys/dev/netif/rl/if_rl.c:1163 vs 1587), so this is a systemic hazard; fixing kern_timeout.c semantics (or auditing all drivers) is the upstream-scale remedy. This finding is filed for if_vr.c per audit scope.

Fix validation

  • fix.diff (vr_stop β†’ callout_stop_async(); vr_detach β†’ callout_drain() moved outside the serializer, before children/resources are torn down): applied to the guest's /usr/src copy, if_vr.ko builds cleanly (PATCHED_BUILD_OK); guest tree reverted afterwards.
  • The remedy's runtime semantics were validated live on the same kernel by module mode 2 (identical call sequence): completes cleanly, callback drained, no wedge, state=2.
  • The real vr(4) hardware path cannot be run (no via-rhine emulation in QEMU 7.2) β†’ fix_status not_testable at the driver level, with compile + pattern validation recorded here.

Honest limits

  • The vr(4) device itself was not attached (no QEMU model); the reproduction is the exact code pattern on the exact kernel primitives, not the NIC irq path. All code-level claims carry path:line citations.
  • The deadlock is a hang, not memory corruption: no escalation attempted; impact ceiling is permanent local DoS (privileged trigger).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

not_testable
baseline reproduced→ patch + rebuild →patched clean

fix.diff (callout_stop_async() in vr_stop + callout_drain() outside the serializer in vr_detach, before child/resource teardown) applies cleanly to the guest /usr/src copy and if_vr.ko builds (PATCHED_BUILD_OK); the identical remedy call-sequence was exercised live via module mode 2 and completes cleanly with the callback drained. The vr(4) hardware path itself cannot be run (QEMU 7.2 has no via-rhine device), so driver-level fix validation is not_testable; compile + runtime-pattern validation recorded.

build.log (PATCHED_BUILD_OK for patched if_vr.ko); run.log/run.2.log dmesg 'poc_deadlock: fixed pattern completed, state=2'; fix.diff
↓ fix.diffper-fix-DF-4260

Confirmed kernel references

Detail

Exploit chain

root: ifconfig vrX down (or kldunload if_vr / shared-irq vr_intr / TX-stall watchdog) races the 1 Hz vr_tick on another CPU -> vr_stop() under if_serializer calls synchronous callout_stop() -> ssleep('costp') holding the serializer while vr_tick blocks acquiring it -> permanent kernel-wide timer death on the tick's CPU; unkillable D-state administrative process. Not memory corruption; no uid0 path.

Evidence (decisive lines)

run.2.log: pid 1705 'D3 costp sysctl -n kern.poc_deadlock=1' + dmesg 'mode 1, tick INPROG on cpu4' (never completes); per-CPU witness 'cpu4 rc=124 elapsed=8s' vs rc=0/3s on cpu0-3,5; run.log: mode 2 'fixed pattern completed, state=2'; VERDICT.md full trace

PoC changes

No seed PoC existed. Module written from scratch: first version used (DFly uses bsd.kmod.mk), a nested /sync/ comment broke the header, and ncpus was already declared in sys/systm.h; all fixed. Witness harness first attempted via cpuset(1) (absent) and plain sleeps (no cpu spread) -- replaced with usched_set(2)-pinned sleeper (4-arg signature).

Verified recommended fix

vr_stop(): callout_stop_async(); vr_detach(): callout_drain() outside if_serializer before tearing down children/resources

Verdict

Deterministically reproduced on the shipping 6.5-DEVELOPMENT SMP guest: calling the synchronous callout_stop() (what vr_stop() does at if_vr.c:1630 under if_serializer on every ifconfig-down/kldunload/intr/watchdog path) while vr_tick()-shaped callback is INPROG-blocked on that serializer permanently wedges the kernel -- stopping thread asleep in 'costp' (kern_timeout.c:920) holding the serializer, one CPU's softclock dead (all timer wakeups on that CPU never fire), sysctl process unkillable. Fixed pattern (callout_stop_async under serializer + callout_drain outside) runs clean; fix.diff compile-checked and pattern-validated.