# DF-5442 — /dev/mem MEMRANGE_SET cross-CPU MTRR race in amd64_mem.c

## What the finding is

`amd64_mrstore()` (sys/platform/pc64/x86_64/amd64_mem.c:280-290) dispatches the
MTRR reprogram function `amd64_mrstoreone()` to the other CPUs with the
**asynchronous, fire-and-forget** `lwkt_send_ipiq_mask()` — it returns as soon
as the remote IPIs are *queued*, without waiting for any remote CPU to execute
them (`sys/kern/lwkt_ipiq.c`, `lwkt_send_ipiq3()`: queues to the target FIFO
and returns; only the sending CPU's own invocation runs inline).

Its only caller `amd64_mrset()` (amd64_mem.c:529-599) then **immediately
rewrites the shared `sc->mr_desc[]` descriptor array** via `amd64_mrfetch()`
(amd64_mem.c:597 → :172-243) — while a remote CPU may still be *reading* that
same array inside `amd64_mrstoreone()` (amd64_mem.c:298-400, reads
`mrd->mr_flags/mr_base/mr_len` per slot).  A remote CPU whose storeone
interleaves with the caller's refetch reads a **torn descriptor generation**
(base from one generation, flags/len from another) and programs a permanent,
per-CPU MTRR pair that matches neither — a cross-CPU memory-type desync
(nothing repairs it: that storeone is the last one queued).  Consequences on
real hardware: one CPU with, e.g., UC or WC where all others have WB over a
physical range → machine check / hang / silent memory corruption.

Nothing else synchronizes the two: the ioctl path holds `mem_lock`
(kern_memio.c:493), but that lock serializes ioctls against *each other*, not
against remote-CPU IPI work.  FreeBSD's counterpart uses the full barrier
`smp_rendezvous_cpus()`; DragonFly's own XXX comments at amd64_mem.c:283-287
and :720-725 admit the missing rendezvous ("We should use ipi_all_but_self()
to call other CPUs into a locking gate").

Reachability: `ioctl("/dev/mem", MEMRANGE_SET, ...)` — root/kmem-gated at open
(devfs 0640 root:kmem + `caps_priv_check(SYSCAP_RESTRICTEDROOT|
__SYSCAP_NOROOTTEST)`, kern_memio.c:152-153).  The missing FWRITE/securelevel
gate on the ioctl itself is the already-filed DF-0080 and is NOT re-reported
here.

## What is in this pack

| file | what it is |
|---|---|
| `mtrr_race.c` | userland PoC (root): control SET/GET/REMOVE roundtrip + race hammer with arrival-time scan |
| `mtrrdump.c` | diagnostic KLD: `debug.mtrr_snap` (per-CPU variable-MTRR snapshot via `lwkt_cpusync_simple` rendezvous + cross-CPU divergence comparator), `debug.mtrr_busy` (busy-IPI load injector), `debug.mtrr_time` (store/fetch cost-center measurement) |
| `Makefile` | stock kmod.mk Makefile (guest tree needs hand-linking, see build.sh) |
| `build.sh` / `run.sh` | exact guest-side build + run |
| `build.log` / `run.log` | compiler + runtime output |
| `logs/final_run.log` | decisive 60-cycle run (canonical) |
| `logs/race.log`, `logs/race2.log` | earlier protocol iterations |
| `logs/env.txt` | guest uname, cpu, sysctls, cost measurements |
| `fix.diff` | the one-line-semantics fix (full rendezvous) |
| `VERDICT.md` | full narrative |

## Build (on the guest, as root)

See `build.sh`.  The guest's `/usr/src` include dirs are trimmed, so the host
pushes `machine.tgz`/`cpu.tgz` (pc64 + x86_64 include trees) and the module is
compiled with the kernel CFLAGS from `sys/platform/pc64/conf/kern.mk`
(`-mcmodel=kernel -mno-red-zone -msoft-float ...`) and linked per
`sys/conf/kmod.mk:207` (`cc -nostdlib -Wl,--hash-style=sysv -r -Wl,-d`).

## Run (on the guest, as root)

`./run.sh` — expected: control roundtrip OK (`A active at desc 89`,
`after REMOVE ... = -1`), then 60 race cycles each printing
`MTRR-SNAP: cpus=6 divergent_slots=0 ...`, then the **detector self-test**
which MUST print `divergent_slots=1` (cpu3 deliberately perturbed) to prove
the comparator works.

## Result on this rig

`not_reproduced` — ~500k MEMRANGE_SET executions across 9 protocol variants
(idle; 2-3 concurrent setter threads; continuous 10-80µs busy-IPI jitter;
caller pinned to cpu0; per-cycle changing final generation; 0-150µs and
0-1000µs arrival-time scans) produced **zero** post-drain cross-CPU MTRR
divergence, with the detector itself proven to fire (self-test) and the phase
costs measured (`var_wrmsr=18 wbinvd=1 var_rdmsr=18 usec`): under KVM the
store/fetch slot loops are phase-locked at the same ~1.1µs/exit cadence, so
the caller's writes and the remote's reads stay per-slot ordered; the tear
needs sub-µs alignment inside a ~2µs band that KVM's exit jitter never lands
in.  On real hardware — where wbinvd costs 10-100µs on large-memory boxes,
making the self-store phase an order of magnitude longer than the fetch phase
— the window is proportionally wider.  The defect itself is code-certain.

Expected output of the decisive run: see `logs/final_run.log`.