DF-4926 — nvmm module unload tears down the hypervisor before the device is dead

nvmm_dragonfly.c destroys the NVMM implementation state while /dev/nvmm is still live, inverting the required destroy_dev()-first order:

• nvmm_detach() (nvmm_dragonfly.c:366-374): unsynchronized nmachines == 0 check at :369, then nvmm_fini() at :372 — which lockuninit()s all 128×128 machine/vcpu locks (nvmm.c:998-1003), runs nvmm_impl->fini() (VMXON regions, MSR bitmaps freed) and sets nvmm_impl = NULL (nvmm.c:1005-1006).
• nvmm_modevent() MOD_UNLOAD (nvmm_dragonfly.c:399-405) calls destroy_dev(dev) only after nvmm_detach() has already done all of that — the device is open and dispatching ioctls the entire time.

Consequences (all reachable by an unprivileged user in group nvmm, given an admin running kldunload nvmm):

1. In-flight ioctl during nvmm_fini(): nvmm_capability() (nvmm.c:245,251) dereferences nvmm_impl on every call → NULL-page kernel deref; any machine ioctl uses lockuninited locks or freed impl state (UAF).
2. Machine-create racing the nmachines check: a machine allocated between :369 and nvmm_fini() survives into destroyed-lock land.
3. Open fd across a successful unload: dfbsd_nvmm_dtor() (nvmm_dragonfly.c:325-335) is a module-text function pointer stored in the file's cdevpriv; closing the fd after unload calls nvmm_kill_machines() through unmapped text.

Build (on the DragonFly guest, as root)

cc -O2 -pthread -o nvmm_race nvmm_race.c

Run

As root, end-to-end (creates the testnv group-nvmm user if missing, builds the racer, loads nvmm, races 40 unload/reload cycles, then a final unload; the racer keeps the fd open for 75 s and closes it at exit):

./run.sh                  # panics are expected; watch the serial log

Note: /dev/nvmm is 0640 root:nvmm, so the unprivileged racer opens it O_RDONLY — DFly does not require FWRITE for ioctls, which is exactly how an unprivileged group-nvmm user drives every NVMM_IOC_* command.

The guest must be booted with the host CPU exposed so nvmm attaches (QEMU_EXTRA="-cpu host" ./vm.sh up).

Expected output

• racer /tmp/race.log: t=1ds cap=... create=... destroy=... counters
• kernel: Fatal trap 12: page fault while in kernel mode with nvmm_capability in the trace (NULL nvmm_impl), or a lockmgr/freed-text panic from the other detonation paths; guest goes down.

DF-4926 VERDICT — REPRODUCED (unload-race kernel crash; fix validated: FIXED)

Finding

sys/dev/virtual/nvmm/nvmm_dragonfly.c tears the NVMM hypervisor down in the wrong order on module unload:

• nvmm_detach() (:366-374): an unsynchronized nmachines == 0 check at :369, then nvmm_fini() at :372 — which lockuninit()s all 128×128 machine/vcpu locks (nvmm.c:998-1003), runs nvmm_impl->fini() (VMXON regions / MSR bitmaps freed) and sets nvmm_impl = NULL (nvmm.c:1005-1006).
• nvmm_modevent() MOD_UNLOAD (:399-405) calls destroy_dev(dev) only after nvmm_detach() has done all of that — the device is open and dispatching ioctls the entire time. destroy_dev() is precisely the operation that revectors ops to dead_dev_ops, drains in-flight calls and blocks new ones (kern/kern_conf.c:334-349, devfs destroy path); it must come FIRST.

Three detonation paths for an unprivileged user in group nvmm (device is crw-r----- root nvmm; DFly ioctls do not require FWRITE, so an O_RDONLY open drives every NVMM_IOC_*):

1. in-flight ioctl during nvmm_fini() → nvmm_impl deref (NULL or freed impl) — e.g. nvmm_capability() nvmm.c:245,251;
2. NVMM_IOC_MACHINE_CREATE landing between the :369 check and nvmm_fini() → machine alive with lockuninited locks and freed impl state (UAF);
3. fd kept open across a successful unload → dfbsd_nvmm_dtor() (:325-335, module text, stored as the cdevpriv dtor at :315) runs nvmm_kill_machines() through unmapped text at close.

Reproduction (2026-09-25, guest DragonFly 6.5-DEVELOPMENT #0 X86_64_GENERIC, backend x86-vmx, QEMU -cpu host)

• Racer (unprivileged testnv, uid 1002, group nvmm, O_RDONLY open): ~1.1M NVMM_IOC_CAPABILITY/s + ~20k machine create/destroy cycles/s (standalone calibration in run.log header; fixed-run totals: 36M caps, 684k cycles).
• Admin side: kldunload nvmm (root) while the racer runs.

Baseline (stock /boot/kernel/nvmm.ko): TWO runs, both panicked the guest with the identical crash site:

Fa[tal trap ...] Stopped at nvmm_ioctl+0x444:  movq 0x48(%rax),%rax

Run 2 panicked on the first kldunload. %rax == nvmm_impl == NULL (just set by nvmm_fini() at nvmm.c:1006 during the in-flight unload); movq 0x48(%rax) is the nvmm_impl->capability function-pointer load in the dispatch (offset 0x48 of struct nvmm_impl, nvmm_internal.h:106-133). Guest down both times; vm.sh status ⇒ down. (The "Fatal trap 12" banner was truncated by the crashing serial console — the Fa fragment — but the faulting RIP/instruction and the dead guest are on the serial log; see panic.txt.)

Patched (fix.diff applied to guest /usr/src, module rebuilt, same race):

• all 40 kldunload attempts during the storm returned EBUSY (rc=1 × 40, rc=0 × 0) — the new nvmm_opens gate held;
• no panic; guest survived the full storm (36M ioctls, 684k machine cycles);
• after the racer died (fd closed → dtor ran against the LIVE module), the final kldunload returned 0 and the module unloaded cleanly.

Impact ceiling

Race-condition kernel memory-safety bug at the hypervisor boundary: the demonstrated result is a NULL-deref panic from unprivileged-steerable ioctls racing an admin unload; paths (2)/(3) give UAF on freed implementation state (VMXON regions, MSR bitmaps, machine/vcpu locks) and execution through unmapped module text — plausible corruption/escalation material for the group-nvmm user, not weaponized here (weaponization needs the admin to run kldunload concurrently, hence Medium rather than High). CVSS: CVSS:3.1/AV:L/AC:H/PR:L/UI:R/S:U/C:N/I:L/A:H (UI:R = the required concurrent kldunload by root).

Fix

fix.diff (git-apply clean against sys/):

1. an nvmm_opens counter (inc at successful open, dec at dtor — after nvmm_kill_machines(), so a concurrent detach observes nmachines==0 before opens==0);
2. nvmm_detach() refuses while any fd is open or any machine exists, then destroy_dev() first, then (stable, drained) re-check with device restore on bail, and only then nvmm_fini();
3. MOD_UNLOAD no longer destroys the dev after the teardown.

Validated end-to-end on the guest (baseline panic vs patched EBUSY/clean unload under the identical storm; race_driver_fixed.log, run_fixed_racer.log, fix_build.log).

PoC changes vs any seed

No executable seed (fresh racer + driver authored). Practical fixes made while bringing it up: su -m testnv -c '... >/tmp/race.log 2>&1 &' runs via root's csh → "Ambiguous output redirect"; replaced with a #!/bin/sh wrapper script executed via su -m testnv -c /home/testnv/goracer.sh &. The racer opens /dev/nvmm O_RDONLY (0640 gives group read only; DFly ioctls need no FWRITE — first attempt with O_RDWR got EACCES).

Guest hygiene

vm.sh reset with-src performed at the end of the run (guest left clean; the testnv user, /root packs and /usr/src patch are on the reverted disk state only).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.