DF-3829 — NVMM VMX: host NMIs swallowed while a vCPU is resident

File under audit: sys/dev/virtual/nvmm/x86/nvmm_x86_vmx.c (pass 2).

What this pack contains

• nvmm_drive.c — raw-ioctl NVMM driver used for the whole pass-2 live verification campaign (exit-dispatcher tests, hostile state, hostile event injection, concurrent stress, vCPU-residency spin). It contains no exploit; it is a correctness/robustness exerciser for the guest->host boundary.
• vctest5.c — minimal machine/hva/gpa/vcpu/setstate reproducer used while debugging the ioctl harness (kept for reference).
• send_nmi.py — sends nmi to the guest via the QEMU monitor socket.
• run.log — full sequential sub-test output (tests A..H).
• run.2.log — full concurrent-stress output (6 vCPUs + 2 hammer threads, ~600k racing ioctls).
• run.seq.log — earlier sequential run (with the accidental garbage-instruction fuzz evidence).
• nmi_ddb_evidence.txt — live-captured serial-console evidence of the NMI trial (guest dropped to DDB from vmx_vcpu_run+0x522).
• env.txt — guest uname, compiler, NMI sysctls, /dev/nvmm mode.
• fix.diff — proposed fix for the swallowed-NMI window.

Build (on the DragonFly guest)

cc -O2 -o nvmm_drive nvmm_drive.c -I/usr/src/sys -lpthread

Prerequisites: kldload nvmm (requires VMX; on the audit guest the VM was booted with qemu-system-x86_64 ... -cpu host and the host KVM has nested=Y), and read access to /dev/nvmm (group nvmm).

Run

# sequential boundary tests (expect: rc=0, no kernel panic)
./nvmm_drive run

# keep 6 vCPUs resident (for the NMI trial) and concurrent stress
./nvmm_drive spin 24 6
./nvmm_drive stress 15 6

# NMI injection from the QEMU monitor (host side)
python3 send_nmi.py

Expected results

• run: every sub-test ends cleanly (HALTED / SHUTDOWN / INVALID), hostile injections are rejected with EINVAL or produce clean VMentry-failure exits, the kernel never panics.
• stress: all threads complete, system alive.
• NMI trial: see nmi_ddb_evidence.txt — under nested KVM the differential for the swallow window cannot be isolated (QEMU nmi hits all CPUs at once); verdict for DF-3829 is inconclusive live, root cause established by source analysis.

DF-3829 — VERDICT

Finding: vmx_exit_exc_nmi() in sys/dev/virtual/nvmm/x86/nvmm_x86_vmx.c (lines 1252–1271) swallows NMI-caused VM exits. When the exit interruption information is a valid NMI, the handler sets exit->reason = NVMM_VCPU_EXIT_NONE and the run loop immediately re-enters the guest (nvmm_x86_vmx.c:2394–2399, 2467–2469). No host-side NMI dispatch happens: the NMI telemetry (DragonFly trap() T_NMI path → isa_nmi() → ddb_on_nmi/panic_on_nmi, sys/platform/pc64/x86_64/trap.c:759–776) is never invoked for an NMI that arrives while the CPU is executing a guest.

Upstream correlation: NetBSD's nvmm_x86_vmx.c added nmi_dispatch() (precisely so the NMI watchdog keeps working while guests run); the DragonFly port (June 2021) predates/omits it and DragonFly has no equivalent API.

Reproduction attempt (live)

Guest: DragonFly 6.5-DEVELOPMENT, 6 vCPUs, QEMU/KVM with -cpu host, host KVM nested=Y; kldload nvmm → "nvmm: attached, using backend x86-vmx" (sys/dev/virtual/nvmm/x86/nvmm_x86_vmx.c backend fully live).

Attack-surface confirmation (relevant context, nvmm_dragonfly.c): /dev/nvmm is root:nvmm 0640. A non-root user in group nvmm cannot open(O_RDWR), but open(O_RDONLY) + full ioctl access works — CAPABILITY, MACHINE_CREATE, HVA_MAP, GPA_MAP, VCPU_CREATE, SETSTATE, RUN all succeed for the unprivileged group member (verified live with user testnv). The entire VMX backend is therefore reachable unprivileged (for group members), confirming the threat model.

Trial: nvmm_drive spin 24 6 (6 vCPUs continuously resident) + 3× QEMU-monitor nmi. Result (serial console, captured before reboot):

CPU0 stopping CPUs: 0x0000003e
 stopped
Stopped at      vmx_vcpu_run+0x522:     movl    %dr0,%edx
db>

The NMI that froze the machine landed while the CPU was in nvmm's host-side run loop (vmx_vcpu_run+0x522 = x86_get_dr0() region, nvmm_x86_vmx.c:2131) — i.e. host-window NMIs are delivered normally. The swallow window (CPU in non-root) cannot be isolated on this guest: QEMU nmi injects into all 6 vCPUs simultaneously, so at least one NMI essentially always lands in a host-side scheduling window and the resulting DDB stop masks the differential. On real hardware (or with a per-CPU NMI source) the differential is: idle → "NMI ... going to debugger"; vCPU resident → nothing, guest keeps running.

Status: inconclusive (live), root cause certain from source. Impact ceiling: while any vCPU is resident on a CPU, hardware-NMI telemetry for that CPU is lost — NMI watchdog, ECC/parity/SERR reporting (isa_nmi() LOG_CRIT paths), and (with panic_on_nmi) intentional panic-on-hardware-failure are suppressed for the residency period. A guest (or an attacker running a CPU-saturating VM) can keep a CPU resident ~100% of the time. No memory-safety impact; availability of diagnostics only. Severity: Low.

Adjacent verification performed during the same run (no findings)

The same campaign exercised, under a live VMX backend, with zero kernel panics:

1. Exit dispatcher end-to-end (vmx_vcpu_run, nvmm_x86_vmx.c:2393– 2461): a real 16-bit guest driven through CPUID (in-kernel handler incl. >max/hypervisor-leaf fallbacks), mov cr0 with CD|NW (in-kernel CR0 handler, nvmm_x86_vmx.c:1646–1703), port-IO exits (simple and, accidentally via mis-encoded guest code, dozens of wild ports/sizes/STR forms — a de-facto fuzz session), RDMSR/WRMSR exits (kernel-handled MSR_IA32_MISC_ENABLE/MSR_TSC/MSR_CR_PAT and user-delivered 0xDEADBEEF), EPT-violation handling (gpa fault-in via nvmm_do_vcpu_run→os_vmspace_fault), HLT/HALTED, triple-fault SHUTDOWN, and VMentry-failure (exit 33) paths. All exits returned well-formed data to the emulator.
2. Hostile state (vmx_vcpu_setstate, 2612–2776): PAT=0xFF…F, EFER=0xD, CR3=0xDEAD000, DR7=0xFF…F → clean VMentry failure (hwcode 33), no panic. FPU state all-0xFF → accepted with MXCSR/MXCSR_MASK sanitization (2762–2764) and the guest ran to HALT — confirming the CVE-2022-30123/30124-family FPU/XCR0 fixes are present and effective (gxcr0 masked at 2674–2677, xstate_bv forced at 2766–2769; the xsave mask vmx_xcr0_mask is only X87|SSE so xsave/xrstor never touch the 576-byte gxsave out of bounds).
3. Hostile event injection (vmx_vcpu_inject, 1125–1181): EXCP vector 0xFF → EINVAL (bounds checks at 1144–1147 live-confirmed); INTR vector 0xFF → clean VMentry failure; EXCP vector 13 with error=0xFFFFFFFF00000000 → accepted without a "VMX fail valid" panic, confirming VMWRITE ignores upper bits for 32-bit VMCS fields (no kernel-reachable vmwrite failure from user error codes).
4. Concurrency: 6 vCPUs resident 15 s + 2 threads performing ~600k racing GETSTATE/SETSTATE/INJECT ioctls against running vCPUs — all succeeded, system alive (vcpu-lock serialization holds).
5. Garbage guest code: wild instruction streams (mis-encoded 32-bit operands in 16-bit mode) produced hundreds of arbitrary exits; every one handled cleanly.

Non-findings recorded for completeness (both analyzed and killed): * Post-exit vmresume returning VMfail (-1 INVALID exits) seen under nested KVM after HLT/entry-failure; logic is identical to NetBSD current, impact is per-vCPU availability of an emulator's own making (possibly a nested-KVM artifact) — not filed. * pmap_add_cpu() called without matching pmap_del_cpu() (XXX comment at 2305–2312) — over-inclusive invalidation masks are conservative-safe. * vmx_htlb_flush/pm_invgen EPT coherence protocol (2206–2236, DragonFly-specific) — reasoned through: double-bump generation + interrupt-forced VMEXIT + pre-launch generation check is sound.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.