DF-5420 — pmap_inval_smp()/pmap_inval_smp_cmpset() single-cpu shortcut samples pmap->pm_active outside the CPULOCK_EXCL interlock (+ early return before target done-bits drain)

What (source-level defects, pass 2)

Three related ordering/protocol weaknesses in the SMP TLB-invalidation engine, all in sys/platform/pc64/x86_64/pmap_inval.c:

1. Shortcut TOCTOU (primary). pmap_inval_smp() line 282 (and pmap_inval_smp_cmpset() line 460) evaluates CPUMASK_CMPMASKEQ(pmap->pm_active, gd->gd_cpumask) before pmap_inval_init() acquires CPULOCK_EXCL (which the shortcut only does at line 289, and only when PMAP_MULTI is set — non-PMAP_MULTI pmaps take no interlock at all). The activation protocol in swtch.s cpu_heavy_restore (OR pm_active bit at :408, then test CPULOCK_EXCL at :410-412, then unconditionally load %cr3 at :445) only guarantees a concurrently-activating cpu is caught if the invalidator samples pm_active under the interlock. Sampling before it (or without it) leaves a window where the activating cpu ORs its bit after the sample, slips its EXCL test in before the acquire, loads %cr3, returns to user mode, and caches the old translation — then the shortcut swaps the PTE and invalidates only the local cpu. That cpu keeps a live TLB entry for a page the kernel has unmapped (use-after-free / info-leak class primitive).
2. pm_active is not a sound "no live TLB" indicator. The full path samples tmpmask = pmap->pm_active at :374 (also :524). A descheduled cpu's bit is cleared at switch-out (swtch.s:200) while its TLB state persists; safety rests entirely on the reactivation %cr3 reload (swtch.s:436-445, deliberately unconditional per the "XXX Temporary cludge" comment — the je optimization is commented out). The invalidation engine hangs PTE teardown correctness on an invariant it does not enforce itself.
3. Early return. The originator returns after clearing only its own done bit (:418-429, :552-558). Targets' trailing invlpgs (executed at pmap_inval_intr :743-767 after the originator's swap) may not have run yet — a real-hardware window where *ptep == 0 but target TLBs still hold the old translation.

Verified on the guest — and the honest outcome

The rig (stale_tlb_race.c) reproduces phantom TLB accesses (successful user-mode load of a va whose PTE is provably 0 — kernel copyin of the same va EFAULTs; the phantom dies at the first syscall) at 0.2–18/min as an unprivileged user. A kernel-instrumentation build (see instrument.diff, ring dumps in instrument_ring_hit.txt) showed the full-path state machine executing correctly (quiesce → originator swap → target invlpg → done bits drain) while hits continued, including under: - machdep.pmap_inval_force_nonopt=1 (shortcut disabled) — 30+ hits, - machdep.pmap_inval_force_allcpus=1 (broadcast to every cpu) — 46 hits, - a fix kernel (fix.diff: interlock hoist + no user-ptep shortcut + broadcast + drain-wait for all target done bits) — 185 hits.

Since the fix kernel guarantees (by construction, with the runs completing i.e. every done bit drained) that every active cpu executed invlpg(va) before munmap() returned, surviving stale entries are architecturally impossible on real hardware — the residual phenomenon is attributable to the virtualization layer (QEMU 7.2/KVM virtual TLB losing guest flushes; host VPID state uninspectable — no host root). The source-level defects above remain real, but could not be demonstrated above this artifact noise floor.

Build

cc -O2 -pthread -o stale_tlb_race stale_tlb_race.c

Run (unprivileged)

./stale_tlb_race 60 1              # 60 s, with stretcher
./stale_tlb_race 60 1 0 1 2        # pinned T0=cpu0 T1=cpu1 Q=cpu2 (root only:
                                   # usched_set(USCHED_SET_CPUMASK) is EPERM
                                   # for unprivileged users; rig continues
                                   # unpinned)

Expected output

Phantom hits print e.g.

[ts] STALE-TLB-HIT type=A(unmapped-load-ok) val=0x... (s1=1 s2=1 ...)
  probe[...]: frozen=1 phase=1 (1=UNMAPPED) kernel-copyin write()=-1 errno=14
  probe[...]: post-syscall user read #1: FAULT

type=A = successful load while the PTE is provably absent (both phase snapshots bracket the load inside the unmapped window). Exit code 42 when any hit was recorded. On real multi-core hardware this rig is expected to be silent on a correct kernel; on this QEMU/KVM guest it fires due to the virtualization artifact described in VERDICT.md — the run is therefore not evidence of a kernel bug by itself.

Files

stale_tlb_race.c     the rig (T0 munmap hammer, T1 sleeping reader + detectors,
                     Q stretcher process, frozen phantom probe)
instrument.diff      kernel instrumentation (branch-level trace of every
                     invalidation of the rig's fixed va) — audit-only
fix.diff             validated-attempt fix (interlock hoist + user-ptep
                     broadcast + drain wait)
run.*.log            baseline / control / probe / instrumented / fixed runs
instrument_ring_hit.txt  kernel ring dump around a hit
env.txt              guest identity and sysctls

DF-5420 — VERDICT

Status: not_reproduced (as a DragonFly kernel defect on the reference guest). The rig reproduces phantom TLB accesses, but exhaustive kernel-side instrumentation and controls attribute the surviving signal to a QEMU/KVM virtual-TLB artifact. The source-level ordering flaws are real and remain likely-confidence hardening findings.

What was run

• Guest: DragonFly 6.5-DEVELOPMENT (master DEV, kernel #0 Thu Jul 2 06:02:54 UTC 2026, identical /usr/src to the audit tree — md5 verified), QEMU 7.2.22
• KVM, -smp 6, machdep.meltdown_mitigation=1.
• Rig: stale_tlb_race.c — unprivileged userland; T0 hammers munmap/mmap/MAGIC writes on a fixed page XADDR; T1 sleeps (jittered 2-80 µs) and guarded-reads XADDR; optional Q stretcher process. Two false-positive- proof detectors (see README).
• 14 recorded runs across stock / sysctl-control / instrumented / fixed kernels (run..log, root.log, fixrun1.log).

Phenomenon (certain, guest level)

Successful user-mode load of XADDR while the PTE is provably 0: phase==PH_UNMAPPED on both sides of the load brackets it inside the munmap→remap window (anon mmap is lazy; the fault-in store happens under PH_WARM/PH_MAPPED), and the frozen probe shows kernel-mode write(XADDR,8) returning -1/EFAULT at the same moment. The phantom dies at T1's next syscall. Rates: 18 hits/60 s (stock, unpriv), 12/60 s (root-pinned), 30+/120 s (shortcut disabled), 46/90 s (all-cpu broadcast), 185/180 s (fix kernel).

Attribution work

1. Shortcut disabled (machdep.pmap_inval_force_nonopt=1) — hits continue → not (only) the shortcut.
2. All-cpu broadcast (machdep.pmap_inval_force_allcpus=1) — hits continue → not (only) the pm_active mask.
3. Kernel instrumentation (instrument.diff: branch-level trace of every invalidation of XADDR — shortcut entry/exit, publish, slave skip/quiesce/wait/invlpg, originator phases, completion). Ring dumps around hits show the full-path state machine operating correctly (quiesce → swap → target invlpg → done drained, e.g. idx 430815-430853 in instrument_ring_hit.txt) while the last command before a dump window was frequently the single-cpu shortcut with pm_active == {cpu0} — the victim cpu was descheduled (bit clear, TLB retained).
4. Exhaustion fix (fix.diff = interlock hoisted before sampling + no shortcut for user-pmap PTE ops + mask = smp_active_mask for those ops + drain-wait until all target done bits clear). Runs completed (no hang ⇒ every done bit drained ⇒ every active cpu executed invlpg(XADDR) before munmap returned) — and hits persisted (185/180 s). On real hardware this configuration makes a surviving entry architecturally impossible (same-cpu program order + IPI-acknowledged invlpg). Therefore the residual signal is lost below the guest architectural level: the QEMU/KVM virtual TLB (VPID-tagged) dropping guest invlpg/mov cr3 flush effects under this cross-vCPU workload. No host root was available to disable/inspect host VPID state or re-run under TCG.

Verdict on the source-level defects

• Leg 1 (shortcut samples pm_active outside the interlock, pmap_inval.c:282/289 and :460; non-PMAP_MULTI takes no interlock at all) is a genuine ordering flaw by protocol construction: the swtch.s activation protocol (OR bit :408 → test EXCL :410 → unconditional cr3 :445) only closes the race if the invalidator samples under EXCL; the full path does (sample at :374 after init at :331), the shortcut does not. Not separately demonstrable on this guest (the artifact noise floor is 2-3 orders of magnitude above the shortcut window's expected hit rate; the instrumented traces never caught a hit whose timing matched the TOCTOU — the victim's bit was consistently long clear, i.e. descheduled, not mid-activation).
• Leg 2 (pm_active ≠ "no live TLB") — real; the protocol leans on the reactivation %cr3 reload (whose same-value load flush is exactly what the virtualization layer drops).
• Leg 3 (early return before target done-bits drain, :418-429/:552-558) — real on any hardware; a µs-scale window with PTE==0 and target TLBs stale; the fix kernel's drain-wait closes it.

Impact assessment

If any leg fires on real hardware: an unprivileged user's thread retains a read/write TLB mapping to a page the kernel has unmapped and freed — a stale-TLB use-after-free primitive (page reuse → cross-process data access; page-table-page reuse → potential privilege escalation). Not demonstrated to uid=0 (no exploitable instance on the reference guest: the observed phantoms die at the first syscall and the attribution is virtualization-side).

Recommended fix

fix.diff (validated to build/boot; efficacy untestable on this guest for the reasons above): hoist pmap_inval_init() before any pm_active sample, drop the single-cpu shortcut for user-pmap PTE-modifying operations, compute their mask from smp_active_mask, and drain target done-bits before returning. The minimal upstream subset (hoist + drain) fixes legs 1 and 3; fully closing leg 2 upstream wants a "recently used this pmap" cpu mask that is not cleared on deschedule (e.g. pm_tlbmask), since trusting the reactivation %cr3 flush has now been shown fragile.

Kernel references

• sys/platform/pc64/x86_64/pmap_inval.c:282-292 (shortcut decision before interlock)
• sys/platform/pc64/x86_64/pmap_inval.c:317-318, 469-474 (PMAP_MULTI-gated init/done)
• sys/platform/pc64/x86_64/pmap_inval.c:374-390 (full-path mask sample)
• sys/platform/pc64/x86_64/pmap_inval.c:418-429, 552-558 (early return, own done bit only)
• sys/platform/pc64/x86_64/pmap_inval.c:460-477 (cmpset shortcut)
• sys/platform/pc64/x86_64/swtch.s:408-445 (activation OR→EXCL test→unconditional cr3)
• sys/platform/pc64/x86_64/swtch.s:176-201 (switch-out clears pm_active bit)

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.