# DF-5420 — pmap_inval_smp()/pmap_inval_smp_cmpset() single-cpu shortcut samples pmap->pm_active outside the CPULOCK_EXCL interlock (+ early return before target done-bits drain)

## What (source-level defects, pass 2)

Three related ordering/protocol weaknesses in the SMP TLB-invalidation engine,
all in `sys/platform/pc64/x86_64/pmap_inval.c`:

1. **Shortcut TOCTOU (primary).** `pmap_inval_smp()` line 282 (and
   `pmap_inval_smp_cmpset()` line 460) evaluates
   `CPUMASK_CMPMASKEQ(pmap->pm_active, gd->gd_cpumask)` **before**
   `pmap_inval_init()` acquires `CPULOCK_EXCL` (which the shortcut only does
   at line 289, and only when `PMAP_MULTI` is set — non-PMAP_MULTI pmaps take
   **no interlock at all**). The activation protocol in `swtch.s`
   `cpu_heavy_restore` (OR pm_active bit at :408, *then* test CPULOCK_EXCL at
   :410-412, *then* unconditionally load %cr3 at :445) only guarantees a
   concurrently-activating cpu is caught if the invalidator samples pm_active
   **under** the interlock. Sampling before it (or without it) leaves a window
   where the activating cpu ORs its bit after the sample, slips its EXCL test
   in before the acquire, loads %cr3, returns to user mode, and caches the old
   translation — then the shortcut swaps the PTE and invalidates **only the
   local cpu**. That cpu keeps a live TLB entry for a page the kernel has
   unmapped (use-after-free / info-leak class primitive).
2. **pm_active is not a sound "no live TLB" indicator.** The full path samples
   `tmpmask = pmap->pm_active` at :374 (also :524). A descheduled cpu's bit is
   cleared at switch-out (`swtch.s:200`) while its TLB state persists; safety
   rests entirely on the reactivation %cr3 reload (`swtch.s:436-445`,
   deliberately unconditional per the "XXX Temporary cludge" comment — the
   `je` optimization is commented out). The invalidation engine hangs PTE
   teardown correctness on an invariant it does not enforce itself.
3. **Early return.** The originator returns after clearing only its **own**
   done bit (:418-429, :552-558). Targets' trailing `invlpg`s (executed at
   `pmap_inval_intr` :743-767 *after* the originator's swap) may not have run
   yet — a real-hardware window where `*ptep == 0` but target TLBs still hold
   the old translation.

## Verified on the guest — and the honest outcome

The rig (`stale_tlb_race.c`) reproduces **phantom TLB accesses** (successful
user-mode load of a va whose PTE is provably 0 — kernel `copyin` of the same
va EFAULTs; the phantom dies at the first syscall) at 0.2–18/min as an
unprivileged user. A kernel-instrumentation build (see `instrument.diff`,
ring dumps in `instrument_ring_hit.txt`) showed the full-path state machine
executing correctly (quiesce → originator swap → target invlpg → done bits
drain) while hits continued, including under:
- `machdep.pmap_inval_force_nonopt=1` (shortcut disabled) — 30+ hits,
- `machdep.pmap_inval_force_allcpus=1` (broadcast to every cpu) — 46 hits,
- a **fix kernel** (fix.diff: interlock hoist + no user-ptep shortcut +
  broadcast + drain-wait for all target done bits) — 185 hits.

Since the fix kernel guarantees (by construction, with the runs completing
i.e. every done bit drained) that every active cpu executed `invlpg(va)`
before `munmap()` returned, surviving stale entries are **architecturally
impossible on real hardware** — the residual phenomenon is attributable to
the virtualization layer (QEMU 7.2/KVM virtual TLB losing guest flushes;
host VPID state uninspectable — no host root). The source-level defects above
remain real, but could not be demonstrated above this artifact noise floor.

## Build

    cc -O2 -pthread -o stale_tlb_race stale_tlb_race.c

## Run (unprivileged)

    ./stale_tlb_race 60 1              # 60 s, with stretcher
    ./stale_tlb_race 60 1 0 1 2        # pinned T0=cpu0 T1=cpu1 Q=cpu2 (root only:
                                       # usched_set(USCHED_SET_CPUMASK) is EPERM
                                       # for unprivileged users; rig continues
                                       # unpinned)

## Expected output

Phantom hits print e.g.

    [ts] STALE-TLB-HIT type=A(unmapped-load-ok) val=0x... (s1=1 s2=1 ...)
      probe[...]: frozen=1 phase=1 (1=UNMAPPED) kernel-copyin write()=-1 errno=14
      probe[...]: post-syscall user read #1: FAULT

`type=A` = successful load while the PTE is provably absent (both phase
snapshots bracket the load inside the unmapped window). Exit code 42 when any
hit was recorded. **On real multi-core hardware this rig is expected to be
silent on a correct kernel; on this QEMU/KVM guest it fires due to the
virtualization artifact described in VERDICT.md — the run is therefore not
evidence of a kernel bug by itself.**

## Files

    stale_tlb_race.c     the rig (T0 munmap hammer, T1 sleeping reader + detectors,
                         Q stretcher process, frozen phantom probe)
    instrument.diff      kernel instrumentation (branch-level trace of every
                         invalidation of the rig's fixed va) — audit-only
    fix.diff             validated-attempt fix (interlock hoist + user-ptep
                         broadcast + drain wait)
    run.*.log            baseline / control / probe / instrumented / fixed runs
    instrument_ring_hit.txt  kernel ring dump around a hit
    env.txt              guest identity and sysctls