vr_stop() invokes the synchronous callout_stop() while holding if_serializer: permanent kernel deadlock (CPU-local timer death) against vr_tick() blocked on that serializer
Summary
vr_tick begins with a blocking lwkt_serialize_enter and re-arms itself; vr_stop calls callout_stop and is always entered with the serializer held. On this kernel callout_stop is synchronous (contradicting its own doc block): when the callback is INPROG it sleeps in 'costp' WHILE THE CALLER STILL HOLDS the serializer the callback needs. Neither side can progress: the stopper sleeps forever holding the serializer, the softclock thread of one CPU wedges acquiring it, all timers on that CPU are dead for the life of the boot, and the triggering process is unkillable. DETERMINISTICALLY REPRODUCED on the 6-CPU guest with a pattern module (sysctl never returns, D3/costp, pinned sleep(3) never wakes on the wedged CPU, collateral oidlk blocks). The identical tick/stop shape exists in other netif drivers (rl, etc.) - the family precedent is DF-3715; this is the vr instance. stop_async + drain-outside fix (validated live, if_vr.ko compile-checked).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4260 Β· 11 files| File | Type | Description | Size | |
|---|---|---|---|---|
| poc_deadlock.c | β | 4.6 KB | view raw | |
| pincpu.c | β | 788 B | view raw | |
| Makefile | β | 60 B | β download | |
| build.sh | β | 174 B | view raw | |
| run.sh | β | 240 B | view raw | |
| build.log | β | 1.6 KB | view raw | |
| run.log | β | 959 B | view raw | |
| run.2.log | β | 874 B | view raw | |
| env.txt | β | 502 B | view raw | |
| fix.diff | β | 1.2 KB | view raw | |
| VERDICT.md | β | 5.9 KB | β raw |
VERDICT β DF-4260
Status: reproduced (deterministic). Impact: dos (permanent kernel deadlock). Guest: DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 2026, x86_64, hw.ncpu=6 (X86_64_GENERIC), fully updated audit snapshot.
What was verified
The vr(4) deadlock pattern was reproduced deterministically on the
shipping guest kernel with a kld module (poc_deadlock.c) that replicates the
exact code shape of if_vr.c β a callout callback that (1) re-arms itself and
(2) lwkt_serialize_enter()s a serializer, plus an administrative path that
holds that same serializer and calls the (synchronous) callout_stop():
kldload poc_deadlock.kosysctl kern.poc_deadlock=2β fixed pattern (callout_stop_async()under the serializer +callout_drain()outside, exactly the remedy in fix.diff): returns cleanly, dmesgfixed pattern completed, state=2.sysctl kern.poc_deadlock=1β bugged pattern (synchronouscallout_stop()under the serializer, i.e. whatvr_stop()at sys/dev/netif/vr/if_vr.c:1630 does on everyifconfig vrX down/kldunload/ interrupt / watchdog path): hangs forever. - Thesysctlprocess is stuck in stateD3, wait channelcostpβ thessleep(c, &c->spin, 0, "costp", 0)in_callout_cancel_or_stop()(sys/kern/kern_timeout.c:920). Unkillable (sleep has no PCATCH);timeout(1)itself blocks forever inwait()(pid 1704pause). - The softclock thread of cpu4 (where the tick was INPROG) is wedged acquiring the serializer held by the sleeping stopper. Witness: ausched_set-pinnedsleep 3returns in 3s on cpu0,1,2,3,5 but never wakes on cpu4 (rc=124 only via timeout's SIGTERM, run.2.log) β cpu4's timer wheel is dead for the life of the boot. - Collateral: any later write to the same sysctl OID blocks forever onoidlk(pid 1735,D5) because the stuck writer holds the OID lock.
Why it happens (code trace)
vr_tick()starts withlwkt_serialize_enter(ifp->if_serializer)(sys/dev/netif/vr/if_vr.c:1165) and re-arms itself (if_vr.c:1178).- softclock marks the callout
CALLOUT_INPROGbefore invoking the callback (sys/kern/kern_timeout.c:566-568), so a callback blocked on the serializer keeps INPROG set. vr_stop()callscallout_stop(&sc->vr_stat_timer)(if_vr.c:1630) and is always called withifp->if_serializerheld:vr_detach(if_vr.c:822-826),vr_ioctl(SIOCSIFFLAGS)β ifioctl dispatchesif_ioctlunderifnet_serialize_all()(sys/net/if.c:2168-2170) β,vr_intr(if_vr.c:1195),vr_watchdog(if_vr.c:1609),vr_init(if_vr.c:1408).- On this kernel
callout_stop()is synchronous (_callout_cancel_or_stop(cc, CALLOUT_STOP, 1), sys/kern/kern_timeout.c:1091-1094): with INPROG set and the caller not being the softclock thread itssleep("costp")s (kern_timeout.c:910-920) while the caller still holds the lwkt serializer the callback needs. lwkt serializers have no owner recursion/wakeup path for a sleeping holder (sys/kern/lwkt_serialize.c:105-111, adaptive spin then sleep) β neither side can ever progress. - The documentation block directly above
callout_stop()still claims it "does not block / does not interlock against a callout that is in-progress" (kern_timeout.c:1086-1093) β the code contradicts it; drivers were written against the documented (async) semantics.callout_stop_async()(kern_timeout.c:1060-1063) preserves the documented behavior.
SMP is required (the callback must be dequeued by another CPU's softclock while the stopper holds the serializer) β satisfied on any multi-core box.
vr(4)-specific reachability
ifconfig vrX down(root) β vr_ioctl β vr_stop under serializer β 1 Hz tick race window of ~Β΅s per attempt (statistical, ~1e-6..1e-4/s while hammering; the vr hardware itself could not be exercised β QEMU 7.2 has no via-rhine PCI device).kldunload if_vr(root) β vr_detach β same.vr_intrβvr_stopwhen IFF_UP was cleared (if_vr.c:1194-1197): the synchronous stop is then executed from the interrupt thread, sleeping in interrupt context while holding the serializer β the nastiest variant.vr_watchdog(TX stall) β vr_stop under serializer (if_vr.c:1609).- The identical
tick-takes-serializer/stop-under-serializershape exists in other DFly netif drivers (e.g. sys/dev/netif/rl/if_rl.c:1163 vs 1587), so this is a systemic hazard; fixing kern_timeout.c semantics (or auditing all drivers) is the upstream-scale remedy. This finding is filed for if_vr.c per audit scope.
Fix validation
fix.diff(vr_stop βcallout_stop_async(); vr_detach βcallout_drain()moved outside the serializer, before children/resources are torn down): applied to the guest's /usr/src copy,if_vr.kobuilds cleanly (PATCHED_BUILD_OK); guest tree reverted afterwards.- The remedy's runtime semantics were validated live on the same kernel by module mode 2 (identical call sequence): completes cleanly, callback drained, no wedge, state=2.
- The real vr(4) hardware path cannot be run (no via-rhine emulation in
QEMU 7.2) β fix_status
not_testableat the driver level, with compile + pattern validation recorded here.
Honest limits
- The vr(4) device itself was not attached (no QEMU model); the reproduction is the exact code pattern on the exact kernel primitives, not the NIC irq path. All code-level claims carry path:line citations.
- The deadlock is a hang, not memory corruption: no escalation attempted; impact ceiling is permanent local DoS (privileged trigger).
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testablefix.diff (callout_stop_async() in vr_stop + callout_drain() outside the serializer in vr_detach, before child/resource teardown) applies cleanly to the guest /usr/src copy and if_vr.ko builds (PATCHED_BUILD_OK); the identical remedy call-sequence was exercised live via module mode 2 and completes cleanly with the callback drained. The vr(4) hardware path itself cannot be run (QEMU 7.2 has no via-rhine device), so driver-level fix validation is not_testable; compile + runtime-pattern validation recorded.
build.log (PATCHED_BUILD_OK for patched if_vr.ko); run.log/run.2.log dmesg 'poc_deadlock: fixed pattern completed, state=2'; fix.diff
Confirmed kernel references
- sys/dev/netif/vr/if_vr.c:1630
- sys/dev/netif/vr/if_vr.c:1165
- sys/dev/netif/vr/if_vr.c:1178
- sys/dev/netif/vr/if_vr.c:822
- sys/dev/netif/vr/if_vr.c:1195
- sys/dev/netif/vr/if_vr.c:1609
- sys/kern/kern_timeout.c:1091
- sys/kern/kern_timeout.c:920
- sys/kern/kern_timeout.c:566
- sys/kern/kern_timeout.c:1086
- sys/net/if.c:2168
- sys/kern/lwkt_serialize.c:105
Detail
Exploit chain
root: ifconfig vrX down (or kldunload if_vr / shared-irq vr_intr / TX-stall watchdog) races the 1 Hz vr_tick on another CPU -> vr_stop() under if_serializer calls synchronous callout_stop() -> ssleep('costp') holding the serializer while vr_tick blocks acquiring it -> permanent kernel-wide timer death on the tick's CPU; unkillable D-state administrative process. Not memory corruption; no uid0 path.
Evidence (decisive lines)
run.2.log: pid 1705 'D3 costp sysctl -n kern.poc_deadlock=1' + dmesg 'mode 1, tick INPROG on cpu4' (never completes); per-CPU witness 'cpu4 rc=124 elapsed=8s' vs rc=0/3s on cpu0-3,5; run.log: mode 2 'fixed pattern completed, state=2'; VERDICT.md full trace
PoC changes
No seed PoC existed. Module written from scratch: first version used
Verified recommended fix
vr_stop(): callout_stop_async(); vr_detach(): callout_drain() outside if_serializer before tearing down children/resources
Verdict
Deterministically reproduced on the shipping 6.5-DEVELOPMENT SMP guest: calling the synchronous callout_stop() (what vr_stop() does at if_vr.c:1630 under if_serializer on every ifconfig-down/kldunload/intr/watchdog path) while vr_tick()-shaped callback is INPROG-blocked on that serializer permanently wedges the kernel -- stopping thread asleep in 'costp' (kern_timeout.c:920) holding the serializer, one CPU's softclock dead (all timer wakeups on that CPU never fire), sysctl process unkillable. Fixed pattern (callout_stop_async under serializer + callout_drain outside) runs clean; fix.diff compile-checked and pattern-validated.
No comments yet.