DF-2721 VERDICT — reproduced (with timing injection); fix validated verbatim

Finding

Lost-wakeup race in the slab allocator's cross-CPU free interlock (z_RSignal/z_RChunks/z_RCount) permanently strands entire slab zones: kernel KVA + wired pages leak forever, invisible to accounting, unbounded, reachable by ordinary unprivileged kernel-object churn on any SMP system.

Root cause (path:line)

_kfree() remote path, sys/kern/kern_slaballoc.c:

• :1511 rsignal = z->z_RSignal; cpu_lfence(); — the remote samples the signal flag before pushing its chunk.
• :1517-1525 — the chunk is pushed onto z->z_RChunks (lock-free CAS).
• :1539 if (bchunk == NULL && rsignal) — the IPI decision uses the stale pre-push sample.

_kmalloc() owner path:

• :991-992 — owner sets z_RSignal=1 only if z_RChunks==NULL, then
• :994 drains, and :1000-1001 TAILQ_REMOVEs the zone from ZoneAry when the last free chunk is handed out.

The interleaving that defeats the interlock:

t0 remote reads z_RSignal == 0 (zone on ZoneAry) t1 owner: swap RSignal=1, drain (RChunks empty), TAILQ_REMOVE (zone leaves ZoneAry; it will never be found by the allocator again) t2 remote pushes its chunk onto z_RChunks (NULL->chunk transition) bchunk==NULL but rsignal==0 -> no IPI is ever sent

Resulting state: zone off ZoneAry, z_NFree==0, z_RChunks != NULL, z_RSignal==1, z_RCount==0. Every recovery path misses it:

• allocation path (:971) only walks ZoneAry;
• slab_cleanup() (:1627) only walks ZoneAry;
• kfree_remote() (:1337) runs only via IPI — none was sent;
• a local free on the owner cpu would re-insert it (:1602-1603) but nothing forces one to happen;
• subsequent remote frees of the same zone read RSignal==1, push with bchunk != NULL and take the else if (rsignal) branch (:1544) — RCount up then down, still no IPI — so every remaining chunk that is freed remotely piles onto the stranded z_RChunks and the entire 128KB zone (32 wired pages + KVA + kernel_map entry) is lost forever;
• accounting shows nothing: the freeing cpu already decremented ks_use[].memuse at :1499-1501.

check_zone_free() cannot recycle the zone because z_NFree can never reach z_NMax (stranded chunks are never drained into z_NFree).

Reproduction (single-tenant QEMU guest, 6 vCPU, DF 6.5-DEVELOPMENT)

Driver: KLD slabstrand.ko — producer lwkt thread pinned cpu0 kmalloc()s 2048-byte chunks (zones owned by cpu0); consumer threads pinned cpu1+cpu2 kfree() them, driving the z_CpuGd != gd remote path at ~51K ops/s/thread; kern.slabstrand.census walks kernel_map counting VM_SUBSYS_KMALLOC entries/KVA; ground-truth metrics: KMALLOC KVA (KB) and v_wire_count.

1. Natural rate bound (stock kernel #0): 5x60s cross-cpu campaigns, 15.4M ops — KMALLOC KVA flat at 82,220KB (zero strands). The precondition (a remote free reading RSignal==0) is common — 51,948 in 30s — but completing the straddle needs the remote to be preempted for the ~30-60ns the owner spends in [swap..remove]; interrupts at ~1-4KHz per cpu put the observed natural rate below 1/17M ops in our window.
2. Injected reproduction (instrumented kernel #1): the ONLY change to protocol behaviour is DELAY(50us) inserted between the rsignal read and the push when a knob is set — a legal emulation of the remote being preempted in that window (kernel_instrumented.c). 6x60s campaign: KMALLOC KVA 80,944 -> 417,200KB (+336MB, monotonic) and v_wire_count +84,647 pages (+330MB wired); after the workload stopped, rings drained, 20s settle: nothing returned (417,200 stays). Hundreds of zones permanently stranded; no panic (stranding is not an INVARIANTS violation — it is a protocol-level leak, not corruption).
3. Fix (same workload, same injection): a. runtime knob implementing the fix decision logic — KVA flat. b. kernel #2 with fix.diff applied byte-for-byte (only the injector knob added for testing; inject=50us now delays every remote free — harsher than (2)): 5.4M remote frees, KVA dead flat at 80,044KB across six 60s runs and after settle; the fix's post-push IPI decision fired 144,525 times and every one was recovered. fix_validation.log.

Impact

Permanent, unbounded, aggregation-only kernel memory leak -> gradual system-wide memory/KVA exhaustion -> kmem_slab_alloc() panic ("kernel_map ran out of space!") / OOM-style denial of service. Trigger: ordinary cross-CPU alloc/free churn (any workload where kernel objects allocated on one cpu are freed on another — migrating threads, interrupt vs syscall cpu splits, network teardown paths). Not memory corruption, not an info leak, not privilege escalation. Natural per-event rate is low (nanosecond window), but each success is permanent and compounds; a patient multi-day workload (or a VM-host jittery environment where virtualization steal-time widens preemption windows by orders of magnitude) accumulates leaks without bound.

The fix (fix.diff — remote side, decision after the push)

Hold z_RCount across the push unconditionally (makes post-push reads of z_RSignal/z_CpuGd safe), then decide the IPI after the push by re-reading z_RSignal: the cpu that performs the NULL->non-NULL z_RChunks transition and sees RSignal set sends the passive IPI regardless of what its stale pre-read said. This closes the window completely: a zone removed from ZoneAry always has RSignal==1, so the transition pusher always notices and kfree_remote() re-attaches the zone. Cost: one extra atomic add/subtract pair per remote free. Validated verbatim on kernel #2 (see fix_validation.log).

Honesty notes

• The first module version double-freed due to a PoC ring bug (two consumers on one SPSC ring); the INVARIANTS kernel caught it immediately ("memory chunk already free!", panic_pocbug_iteration.txt). Fixed (per-consumer rings); the final sources in this pack are clean.
• The hits counter on instrumented kernel #1 (post-push RSignal==1 with pre-read==0) has false positives (stale RSignal==1 is legal on on-list zones), which is why the pack's ground truth is the merge-immune KMALLOC-KVA / wire-count census, not the counter.
• Natural (un-injected) reproduction was NOT observed in ~17M ops; all massive-leak demonstrations use the 50us injector. The injector only widens an existing window (preemption between two adjacent operations); it does not change the protocol or force any state.

Guest hygiene

Guest reset to clean with-src snapshot after this run.