DF-5442 — /dev/mem MEMRANGE_SET cross-CPU MTRR race in amd64_mem.c

What the finding is

amd64_mrstore() (sys/platform/pc64/x86_64/amd64_mem.c:280-290) dispatches the MTRR reprogram function amd64_mrstoreone() to the other CPUs with the asynchronous, fire-and-forget lwkt_send_ipiq_mask() — it returns as soon as the remote IPIs are queued, without waiting for any remote CPU to execute them (sys/kern/lwkt_ipiq.c, lwkt_send_ipiq3(): queues to the target FIFO and returns; only the sending CPU's own invocation runs inline).

Its only caller amd64_mrset() (amd64_mem.c:529-599) then immediately rewrites the shared sc->mr_desc[] descriptor array via amd64_mrfetch() (amd64_mem.c:597 → :172-243) — while a remote CPU may still be reading that same array inside amd64_mrstoreone() (amd64_mem.c:298-400, reads mrd->mr_flags/mr_base/mr_len per slot). A remote CPU whose storeone interleaves with the caller's refetch reads a torn descriptor generation (base from one generation, flags/len from another) and programs a permanent, per-CPU MTRR pair that matches neither — a cross-CPU memory-type desync (nothing repairs it: that storeone is the last one queued). Consequences on real hardware: one CPU with, e.g., UC or WC where all others have WB over a physical range → machine check / hang / silent memory corruption.

Nothing else synchronizes the two: the ioctl path holds mem_lock (kern_memio.c:493), but that lock serializes ioctls against each other, not against remote-CPU IPI work. FreeBSD's counterpart uses the full barrier smp_rendezvous_cpus(); DragonFly's own XXX comments at amd64_mem.c:283-287 and :720-725 admit the missing rendezvous ("We should use ipi_all_but_self() to call other CPUs into a locking gate").

Reachability: ioctl("/dev/mem", MEMRANGE_SET, ...) — root/kmem-gated at open (devfs 0640 root:kmem + caps_priv_check(SYSCAP_RESTRICTEDROOT| __SYSCAP_NOROOTTEST), kern_memio.c:152-153). The missing FWRITE/securelevel gate on the ioctl itself is the already-filed DF-0080 and is NOT re-reported here.

What is in this pack

Build (on the guest, as root)

See build.sh. The guest's /usr/src include dirs are trimmed, so the host pushes machine.tgz/cpu.tgz (pc64 + x86_64 include trees) and the module is compiled with the kernel CFLAGS from sys/platform/pc64/conf/kern.mk (-mcmodel=kernel -mno-red-zone -msoft-float ...) and linked per sys/conf/kmod.mk:207 (cc -nostdlib -Wl,--hash-style=sysv -r -Wl,-d).

Run (on the guest, as root)

./run.sh — expected: control roundtrip OK (A active at desc 89, after REMOVE ... = -1), then 60 race cycles each printing MTRR-SNAP: cpus=6 divergent_slots=0 ..., then the detector self-test which MUST print divergent_slots=1 (cpu3 deliberately perturbed) to prove the comparator works.

Result on this rig

not_reproduced — ~500k MEMRANGE_SET executions across 9 protocol variants (idle; 2-3 concurrent setter threads; continuous 10-80µs busy-IPI jitter; caller pinned to cpu0; per-cycle changing final generation; 0-150µs and 0-1000µs arrival-time scans) produced zero post-drain cross-CPU MTRR divergence, with the detector itself proven to fire (self-test) and the phase costs measured (var_wrmsr=18 wbinvd=1 var_rdmsr=18 usec): under KVM the store/fetch slot loops are phase-locked at the same ~1.1µs/exit cadence, so the caller's writes and the remote's reads stay per-slot ordered; the tear needs sub-µs alignment inside a ~2µs band that KVM's exit jitter never lands in. On real hardware — where wbinvd costs 10-100µs on large-memory boxes, making the self-store phase an order of magnitude longer than the fetch phase — the window is proportionally wider. The defect itself is code-certain.

Expected output of the decisive run: see logs/final_run.log.

DF-5442 VERDICT — not_reproduced (on this KVM guest); defect code-certain

The defect (path:line)

• sys/platform/pc64/x86_64/amd64_mem.c:289 — amd64_mrstore() dispatches amd64_mrstoreone() to all CPUs with the asynchronous lwkt_send_ipiq_mask(); remote CPUs merely get a FIFO entry (sys/kern/lwkt_ipiq.c lwkt_send_ipiq3(), queue-and-return; only the sender's own invocation runs inline).
• sys/platform/pc64/x86_64/amd64_mem.c:594-597 — amd64_mrset() calls amd64_mrstore(sc) and then immediately amd64_mrfetch(sc), which rewrites every entry of the shared sc->mr_desc[] array (:187-241) from the caller's MSRs — with zero synchronization against a remote CPU still executing amd64_mrstoreone() (:306-386), which walks and reads that same array slot-by-slot (mrd->mr_flags, mrd->mr_base, mrd->mr_len).
• The mem_lock held by the ioctl path (sys/kern/kern_memio.c:493) only serializes ioctls against each other; it says nothing about remote-CPU IPI execution.
• The same pattern in amd64_mrreinit() (:726) for AP re-init, and amd64_mrAPinit() reached from mem_range_AP_init() (sys/platform/pc64/x86_64/mp_machdep.c:1521) during AP boot.
• FreeBSD's counterpart reprograms MTRRs under smp_rendezvous_cpus() — a full barrier that returns only after every CPU finished. DragonFly's own XXX comments at :283-287 and :720-725 admit the missing gate.

A remote CPU whose storeone interleaves with the caller's refetch reads a torn (base, flags, len) triple and programs an MTRR pair that matches neither generation — and because that storeone is the last one queued, nothing ever repairs it: the desync persists on that CPU (e.g. slot programmed with base=0 + mask=VALID only = whole physical address space UC, from mixing an inactive slot's stale base/len with the new ACTIVE flags). On real hardware that is machine-check / hang / silent-corruption material for whatever physical ranges end up mistyped on one CPU only.

Entry: ioctl("/dev/mem", MEMRANGE_SET, ...) — root/kmem-gated at open (kern_memio.c:140-160). The missing FWRITE/securelevel gate on the ioctl itself is already filed as DF-0080 and is not re-reported.

Verification effort (9 protocol variants, ~500k SETs)

Rig: 6-vCPU KVM guest (DragonFly 6.5-DEVELOPMENT, kernel of Jul 2 2026), ndesc=96 (88 fixed + 8 variable MTRRs), MEMRANGE_SET/GET/REMOVE roundtrip verified (control: A active at desc 89, REMOVE → -1).

Observation tool: mtrrdump.ko — debug.mtrr_snap reads all 8 variable-MTRR base/mask MSR pairs plus MTRRdefType on every CPU inside a lwkt_cpusync_simple() rendezvous (which also drains each CPU's pending IPI FIFO before sampling), then compares across CPUs; debug.mtrr_busy fires a µs-spin IPI at all CPUs (load injector); debug.mtrr_time measures the cost centers.

Detector validated: a deliberately perturbed build (cpu3 slot0 ^= 0x1000) reports divergent_slots=1 with the per-CPU table; the clean build reports 0.

Protocols tried: 1. single-threaded alternation, idle system (550 SETs) — 0 divergence; 2. 2-3 concurrent setter threads (~110k SETs) — 0; 3. continuous 60-120µs busy-IPI jitter (~68k SETs) — 0; 4. caller pinned to cpu0 + per-cycle changing final generation (~67k SETs) — 0; 5. busy injectors live through the final SET (~62k SETs) — 0; 6. arrival-time scan, busy 0-150µs in 10µs steps before each final SET (~104k SETs) — 0; 7. arrival-time scan 0-1000µs in 50µs steps (~81k SETs) — 0; 8. final canonical run (60 cycles, ~14k SETs) — 0, guest healthy throughout; 9. plus the ENOSPC-triggering variant (slots exhausted) that proved the ENOSPC path at amd64_mem.c:515 behaves as read.

Why it does not reproduce under KVM (measured)

debug.mtrr_time: var_wrmsr=18 wbinvd=1 var_rdmsr=18 usec (tsc_frequency=3.6GHz). So T_s (storeone) ≈ 32µs and T_f (mrfetch) ≈ 30µs, with both slot loops advancing at the same ~1.1µs/MSR-exit cadence. Caller-side writes of slot j and remote-side reads of slot j are therefore offset by the constant (T_s + fixed_f + j·c) − (ε + fixed_r + j·c) = ~30µs−ε: either the whole remote read pass happens before the caller's write pass or after it. A tear requires the sub-µs field-store/load interleaving inside a ~2µs-wide arrival band, under KVM exit jitter that randomizes each trial past the band. ~500k trials, zero hits — with the detector proven to fire.

On real hardware the geometry differs materially: wbinvd costs 10-100µs on large-memory systems (vs 1µs here), which lengthens the caller's self-store phase and widens the absolute misalignment distribution (real interrupt latency, SMIs, other IPI work), making the overlap reachable. The defect is unconditional in the code; only its timing window is guest-hostile here.

Verdict

status: not_reproduced on this virtualization stack (expected marker — post-drain divergent_slots>0 — absent across ~500k SETs and 9 protocols; guest stayed up; detector self-test proves the instrumentation). The finding itself stands on code analysis with confidence likely: unsynchronized shared-state access is present by construction; the manifestation needs timing this KVM rig does not deliver.

Fix validation

Not performed (fix_status: not_testable): with no reproducible baseline on this guest there is no observable behavior to compare a patched kernel against. fix.diff swaps the two fire-and-forget dispatches for lwkt_cpusync_simple() — the exact primitive the in-tree XXX comments ask for — verified git apply --check clean against the audit tree; a patched kernel additionally makes each MEMRANGE_SET wait for all CPUs (measurable as a latency change, not a behavioral fix observable by this rig).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.