PoC DF-4529 — nvme(4) detach vs in-flight NVMEIOCGETLOG: use-after-free → kernel div-by-zero panic (REPRODUCED)

What this proves

nvme_pci_detach() tears down and frees the admin queue while user threads holding an open namespace-device descriptor can still be inside nvme_getlog_ioctl() → nvme_get_admin_request() → nvme_get_request(). Nothing blocks new ioctls, drains in-flight ones, or takes sc->ioctl_lk on the detach side (NVME_SC_UNLOADING is checked in nvme_open() but is never set anywhere; sc->opencnt is tracked but never consulted).

When the race hits, nvme_free_subqueue() (nvme.c:298) has bzero()'d the submission queue (nqe == 0) while the ioctl thread evaluates the ring-full check in nvme_get_request():

if ((queue->subq_tail + queue->unsubmitted + 1) % queue->nqe ==   /* nvme.c:446 */
    queue->subq_head) {

% queue->nqe with nqe == 0 → integer divide fault → kernel panic. Observed twice (deterministic with the rig below):

Fatal trap 18: integer divide fault while in kernel mode
instruction pointer = 0x8:0xffffffff80310904
current process      = 934        (an /root/nvspam spammer)
Stopped at nvme_get_request+0x34: divl 0x40(%r13),%eax

0x40(%r13) is queue->nqe (offset 0x40 in struct nvme_subqueue, nvme.h:118-155). panic.txt holds the full trap block from the serial console.

Environment

• Private QEMU/KVM instance of the audit guest (dfbsd-qemu/vm_nvme.sh): the stock virtio-boot guest plus -device nvme,id=nv0,serial=AUDNVME, msix_qsize=65,max_ioqpairs=8 and 8 -device nvme-ns namespaces backed by null-co blockdevs (the guest's stock config has no NVMe device).
• hw.nvmebang.action test module (nvmebang.c, in this pack) invokes the nvme driver's own attach/detach driver-methods on unit 0 on demand — a deterministic, repeatable stand-in for PCI hot-unplug / surprise removal, which runs the identical nvme_pci_detach() path.
• nvspam (root): loops ioctl(NVMEIOCGETLOG) on /dev/nvme*, i.e. the raw-admin-command ioctl, ~10-25k calls/s per process.

Build

In the guest (as root):

# spammer + instrumented probers
cc -O2 -o /root/nvspam  nvspam.c
cc -O2 -o /root/nvslow  nvslow.c
cc -O2 -o /root/nvgetlog nvgetlog.c
# trigger module
cd /root/poc/bang   # Makefile: SYSDIR=/usr/src/sys, SRCS=nvmebang.c
                   #           device_if.h bus_if.h pci_if.h
make && kldload ./nvmebang.ko

Run (exact rig)

# 6 spammers hammering the admin ioctl path
for i in 1 2 3 4 5 6; do /root/nvspam > /root/sp$i.log 2>&1 & done
sleep 5
# one device removal
sysctl hw.nvmebang.action=1

Expected

Unfixed kernel: panic within seconds — Fatal trap 18 / nvme_get_request+0x34 divl 0x40(%r13),%eax (see panic.txt; reproduced 2/2 fresh boots, first detach each time).

Patched kernel (fix.diff applied, kernel rebuilt): nvme0: detached in dmesg, no panic, spammers keep running and see only graceful ioctl errors (ENXIO) afterwards. Repeated detach/attach cycles stay clean.

VERDICT — DF-4529

Status: reproduced (impact: panic — use-after-free-driven kernel divide-by-zero). Reproduced 2/2 on fresh boots; fix validated by kernel rebuild: panic gone, detach clean.

Root cause

1. nvme_open() (nvme_disk.c:132) checks sc->flags & NVME_SC_UNLOADING — but nothing in the tree ever sets NVME_SC_UNLOADING (grep NVME_SC_UNLOADING sys/dev/disk/nvme/*.c → only the check).
2. nvme_getlog_ioctl() (nvme_ioctl.c:48-60) holds sc->ioctl_lk for its whole nvme_get_admin_request → submit → wait → put sequence.
3. nvme_pci_detach() (nvme_attach.c:470-517) never takes sc->ioctl_lk: it stops the admin thread, destroys the disk (cdevs survive while descriptors stay open), frees and bzero()s the namespace softcs (kfree(nsc) in nvme_stop_admin_thread, nvme_admin.c:117-126), then nvme_free_subqueue(sc, 0) (nvme.c:273-299) kfree()s reqary and bzero()s the whole submission queue — queue->nqe = 0 — and nvme_free_comqueue(sc, 0) (nvme.c:301-317) frees the completion ring.
4. A user thread whose ioctl already dispatched (descriptor still open; devfs keeps the cdev alive for open fds) — or that dispatches through the still-resolvable si_drv1 — executes nvme_get_request() with the freed/bzero'd queue: (queue->subq_tail + queue->unsubmitted + 1) % queue->nqe (nvme.c:446) with nqe == 0 → integer divide fault. Even when the div0 window is missed, the ioctl path reads and writes freed memory (req->res, nvme_put_request()'s cmpset on queue->first_avail, nvme.c:617-623) — a silent UAF into the kmalloc free pool.

Reproduction

Rig: 6 × nvspam (root) looping NVMEIOCGETLOG (~100k calls/s total) against /dev/nvme0..7; sysctl hw.nvmebang.action=1 invokes the nvme driver's own detach driver-method (the exact code PCI hot-unplug / surprise removal runs, made deterministic and repeatable).

• Run 1 (12 spammers): Fatal trap 18 ... nvme_get_request+0x34: divl 0x40(%r13),%eax, current process = 934 (spammer).
• Run 2 (fresh boot, 6 rc.local spammers): identical trap on the first detach, current process = 633.

Full trap block: panic.txt. Guest stayed in DDB (db>) after the panic (fatal traps stop the machine; no dump was configured).

Determination of 0x40(%r13) = nqe: offset 0x40 in struct nvme_subqueue (nvme.h:118-155: lock 0x28 … nqe at 0x40, qid 0x44 …); divl is the modulo in the ring-full check — the only division in nvme_get_request().

Why a privileged trigger does not cap this at "not a security bug"

The detach side is a legitimate administrative/hardware event (hot-unplug, surprise removal, VM device_del, chassis maintenance) that must be safe while the device is open — the kernel corruptating itself (divide-by-zero on freed data; silent writes into the freed reqary) is a kernel robustness/security defect regardless of who triggered the removal. The ioctl side needs an open raw-disk fd (root, or an operator-group user per the node's 0640 — note DF-2217's claim of operator reach was not reproducible here: an operator-group test user got EPERM on open("/dev/nvme0") even before the ioctl, so the observed rig used root).

Fix

fix.diff (validated):

• nvme_pci_detach(): set NVME_SC_UNLOADING; hold sc->ioctl_lk exclusively across the admin-thread stop and the queue-0 teardown so every in-flight ioctl has returned before any free.
• nvme_getlog_ioctl(): refuse new commands with ENXIO once unloading.
• nvme_stop_admin_thread(): NULL nsc->sc before kfree(nsc) so late dispatches through a still-open fd fail cleanly.
• nvme_disk.c nvme_ioctl(): NULL-guard nsc->sc.

Baseline (unfixed): panic on first detach under load (2/2). Patched: rebuild + install + reboot, same rig (6 spammers + detach, then repeated detach/attach cycles): no panic, nvme0: detached logged, spammers continue and receive ENXIO/EIO errors only. See fix_build.log / fix_run.log.

Kernel references

• sys/dev/disk/nvme/nvme.c:446 (ring-full % nqe — div0 site)
• sys/dev/disk/nvme/nvme.c:273-299 (nvme_free_subqueue kfree+bzero)
• sys/dev/disk/nvme/nvme_ioctl.c:48-60 (unserialized ioctl side)
• sys/dev/disk/nvme/nvme_attach.c:470-517 (detach w/o ioctl_lk / drain)
• sys/dev/disk/nvme/nvme_disk.c:132 (dead NVME_SC_UNLOADING check)
• sys/dev/disk/nvme/nvme_admin.c:117-126 (nsc kfree, si_drv1 dangling)

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.