DF-2721 / verdict.json
{ "finding_id": "DF-2721", "status": "reproduced", "reproduced": 1, "impact": "dos", "confidence": "certain", "verdict": "Lost-wakeup race in the slab allocator remote-free interlock (z_RSignal pre-read at kern_slaballoc.c:1511 vs IPI decision at :1539 racing the owner's zone removal at :991-1001) permanently strands whole 128KB slab zones with their remotely-freed chunks in z_RChunks: no IPI is ever sent, the zone leaves ZoneAry, and no recovery path (alloc path :971, slab_cleanup :1627, kfree_remote :1337, local free :1602) can ever reach it again. Each stranded zone permanently leaks 128KB KVA + 32 wired pages + a kernel_map entry, invisible to accounting. Reproduced on the 6-vCPU guest with a pinned-thread KLD hammer: natural window too rare to observe in 15.4M ops (precondition - reading RSignal==0 - fires on ~1/30 remote frees, 51,948/30s, but the ~50ns straddle never completed); with a 50us preemption injected between the two adjacent remote-side operations (a legal interleaving any interrupt can produce) the leak is massive and permanent: +336MB KMALLOC KVA and +330MB wired pages over 6 minutes, none returned after drain+settle. Fix (decide the IPI AFTER the push by re-reading z_RSignal under an unconditionally-held z_RCount) validated twice, including on a kernel with fix.diff applied byte-for-byte: 5.4M injected remote frees, zero net KVA/wire growth, 144,525 recovered would-be strands. Impact ceiling: unbounded permanent kernel memory leak -> gradual memory exhaustion DoS from unprivileged cross-CPU churn; not corruption, not info disclosure, not privesc.", "exploit_chain": "unprivileged workload causing cross-cpu kmalloc/kfree churn (migrating threads, irq-vs-syscall cpu split) -> remote kfree reads z_RSignal==0 -> owner exhausts zone (swap RSignal=1, drain, TAILQ_REMOVE) inside the remote's read..push window (naturally ~50ns, widened by any preemption/virtualization steal) -> remote pushes chunk, stale rsignal==0, no IPI -> zone permanently off ZoneAry with NFree==0 -> all subsequent remote frees of that zone strand too (bchunk!=NULL path never IPIs) -> entire zone (128KB KVA, 32 wired pages, kernel_map entry) leaked forever -> aggregate unbounded -> eventual kmem_slab_alloc panic/OOM DoS.", "evidence": [ "run.log sections [1]-[5]: stock flat at 82220KB over 15.4M ops; instrumented+50us inject: kmalloc_kva_kb 80944->417200 (+336MB) and wire +84647 pages, retained after settle", "fix_validation.log: kernel #2 with fix.diff verbatim, same injection on all 5.4M remote frees: kmalloc_kva_kb dead flat 80044KB, hits(fix decisions)=144525 all recovered", "VERDICT.md: full interleaving analysis with path:line for every recovery path that misses the stranded zone", "kernel_instrumented.c / kernel_fixed.c / fix.diff: test kernel, fixed kernel, production fix", "panic_pocbug_iteration.txt: honesty note - first PoC iteration's own double-free caught by INVARIANTS" ], "kernel_refs": [ "sys/kern/kern_slaballoc.c:1511", "sys/kern/kern_slaballoc.c:1517", "sys/kern/kern_slaballoc.c:1539", "sys/kern/kern_slaballoc.c:991", "sys/kern/kern_slaballoc.c:1000", "sys/kern/kern_slaballoc.c:1337", "sys/kern/kern_slaballoc.c:1602", "sys/kern/kern_slaballoc.c:1627" ], "poc_changes": "Original sketch was syscall-driven (too slow: ~1K chunks/s); rewrote as in-kernel KLD with lwkt threads pinned via TDF_FIXEDCPU (51K+ ops/s/thread). First version double-freed (two consumers, one SPSC ring) - INVARIANTS caught it; final version uses one SPSC ring per consumer. Added kernel_map census sysctl (VM_SUBSYS_KMALLOC entries/KVA) as merge-immune ground truth after zone-count noise from adjacent-entry merging. Injector (DELAY in the legal preemption window) added to the guest's kernel copy only.", "attempts": 6, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #0 X86_64_GENERIC (stock repro attempts); #1 instrumented (injected reproduction); #2 fix.diff-verbatim (fix validation) - all Thu Jul 2/Mon Aug 31 2026 builds, 6 vCPU, 4GB", "runtime_sec": 4200, "guest_dirty": 1, "build_cmd": "cd ~/slabstrand && make (KLD) ; kernel: cd /usr/src && make -j6 nativekernel KERNCONF=X86_64_GENERIC && make installkernel KERNCONF=X86_64_GENERIC", "run_cmd": "kldload /tmp/slabstrand.ko; sysctl -w debug.slab_strand_watch=1; sysctl -w debug.slab_strand_inject=50; sysctl -n kern.slabstrand.krun=60 (x6); sysctl kern.slabstrand.census", "code_hash": "15582deefaf76ec3c47ab7f4252496db95136d17e621ff68771c17c78f6056dd", "notes": "Natural (un-injected) rate < 1/17M ops observed - the defect is real and permanent-per-hit but each natural hit is rare; virtualization/steal-time environments widen the window materially. leak is NOT visible in vmstat -m (memuse already decremented); only in kernel_map size / v_wire_count. PoC requires root only to LOAD the driver; the underlying workload pattern is ordinary unprivileged syscall churn.", "recommended_fix": "In _kfree() remote path: hold z_RCount across the RChunks push unconditionally and make the IPI decision AFTER the push by re-reading z_RSignal (IPI on the NULL->non-NULL transition when RSignal is set) - see fix.diff.", "fix_status": "fixed", "fix_kernel_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #2: Mon Aug 31 08:53:06 UTC 2026 (fix.diff applied verbatim + injector knob only)", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "Same injected 6x60s campaign on the fix.diff-verbatim kernel: kmalloc_kva_kb flat at 80044KB across all runs and after settle (vs +336MB on stock protocol); wire flat; the fix decision fired 144525 times under injection with every zone recovered; throughput unchanged (~900K ops/run).", "fix_evidence": "fix_validation.log (campaign4); kernel_fixed.c; fix.diff" } |