DF-4818 — nvme(4): unchecked MSI-X vector IRQ resource allocation + missing

MSI-X teardown on fallback (nvme_pci_attach)

What this is

nvme_pci_attach() (sys/dev/disk/nvme/nvme_attach.c:149-205) requests one SYS_RES_IRQ per MSI-X vector:

160:  for (i = 0; error == 0 && i < sc->nirqs; ++i) {
162:      error = pci_alloc_msix_vector(dev, i, &sc->rid_irq[i], cpu);
164:      if (error)
165:          break;
166:      sc->irq[i] = bus_alloc_resource_any(dev, SYS_RES_IRQ,
167:                                          &sc->rid_irq[i], RF_ACTIVE);

The return of bus_alloc_resource_any() is never checked, and after the loop only sc->irq[0] is validated (line 239). Three consequences, all boot/kldload-time kernel panics:

1. NULL irq reaches bus_setup_intr — if bus_alloc_resource_any fails for vector k>=1 (kernel memory pressure in rman_reserve_resource, or rman/map desync) while vector 0 succeeds, attach completes with sc->irq[k] == NULL. The admin thread later runs nvme_create_comqueue() which calls bus_setup_intr(sc->dev, sc->irq[ivect], ...) (nvme.c:823) for ivect == qid; on NULL the call chain reaches nexus_setup_intr() which does an unconditional panic("nexus_setup_intr: NULL irq resource!") (sys/platform/pc64/x86_64/nexus.c:504-506).

2. NULL irq hits the error unwind — if instead a LATER vector fails in pci_alloc_msix_vector after an earlier vector's resource alloc returned NULL, the unwind loop (nvme_attach.c:192-199) calls bus_release_resource(dev, SYS_RES_IRQ, rid, NULL) → resource_list_release() → unconditional panic("resource_list_release: resource entry is not busy") (sys/kern/subr_bus.c:2738-2739).

3. Missing pci_teardown_msix on the MSI-X→legacy fallback — when pci_setup_msix() succeeded but a vector allocation then fails, the fallback path (line 225) overwrites sc->irq_type, and detach (line 508-517) only tears MSI-X down for irq_type == PCI_INTR_TYPE_MSIX. msix_table_res/msix_pba_res stay set, so a later re-attach of the same device hits KASSERT(... "MSI-X has been setup yet") in pci_setup_msix() (sys/bus/pci/pci.c:1644-1645). INVARIANTS kernels only — which includes the GENERIC config of this audit (sys/config/X86_64_GENERIC:56), as proven live by the unrelated vtpci KASSERT panic captured in run.3.log.

How to build / run

There is no C PoC: the trigger is an environmental failure inside the kernel resource allocator at driver-attach time. The evidence pack instead live-validates every reachable sub-path of the MSI-X error machinery on the QEMU guest, bracketing the defect to exactly the unchecked NULL:

1. ./run.sh baseline — boot dfbsd-qemu/vm_nvme.sh (1 nvme controller, msix_qsize=65, -smp 6): expect nvme0: mapped 7 MSIX IRQs (= min(65, ncpus+1)).
2. ./run.sh emptypool — add hw.ioapic.msi_start="192" to /boot/loader.conf and reboot: per-cpu MSI pool empty → every pci_alloc_msix_vector fails at i=0 → unwind no-op → legacy fallback: nvme0: mapped 1 normal-int IRQs, disk still attaches. (run.2.log)
3. ./run.sh midloop — dfbsd-qemu/vm_nvme24.sh (24 controllers, -smp 2) + hw.ioapic.msi_start="160" (pool 2x32=64 slots vs 243=72 demanded): nvme21 fails MID-loop, the unwind loop runs with a live allocated vector, releases it, falls back; nvme21..23 attach their disks; no nvme panic*. (run.3.log)

The panic itself (manifestations 1-3) could not be forced on the emulator: manifestation 1/2 need rman_reserve_resource to fail on a freshly granted MSI-X irq number (kernel-memory pressure / desync — the per-cpu map and rman are derived from the same source, sys/platform/pc64/apic/ioapic_abi.c:1200- 1245), and manifestation 3 needs a re-attach cycle of a driver compiled into the kernel (no sys/modules/nvme exists).

Expected output

See run.log (baseline), run.2.log (empty pool), run.3.log (mid-loop + the incidental vtpci panic — a different driver's bug, see VERDICT.md).

DF-4818 VERDICT — nvme_attach.c MSI-X error path

Status: INCONCLUSIVE (code-certain defect; panic trigger not forceable on the emulator; all reachable sub-paths exercised live and proven safe). Expected impact if triggered: kernel panic (boot / driver-load-time DoS).

Root cause (code-proven)

• sys/dev/disk/nvme/nvme_attach.c:166-168 — sc->irq[i] = bus_alloc_resource_any(...) return value never checked; error only set by pci_alloc_msix_vector.
• sys/dev/disk/nvme/nvme_attach.c:239 — only sc->irq[0] is validated.
• sys/dev/disk/nvme/nvme_attach.c:192-199 — unwind passes sc->irq[i] (possibly NULL) straight into bus_release_resource.
• sys/dev/disk/nvme/nvme_attach.c:225-238 — MSI-X→legacy fallback does not call pci_teardown_msix() after a successful pci_setup_msix(); sc->irq_type is overwritten so detach (lines 508-517) skips teardown.

Sink proofs

1. NULL-irq → panic: nvme_create_comqueue (sys/dev/disk/nvme/nvme.c:823) calls bus_setup_intr(sc->dev, sc->irq[ivect], ...) with the unchecked pointer → pci_setup_intr (sys/bus/pci/pci.c:3285) → bus_generic_setup_intr (sys/kern/subr_bus.c:3019) → nexus_setup_intr (sys/platform/pc64/x86_64/nexus.c:498-506): if (irq == NULL) panic("%s: NULL irq resource!", __func__); — an unconditional panic(), not an assert.
2. NULL-irq in unwind → panic: bus_release_resource(..., NULL) → bus_generic_rl_release_resource → resource_list_release (sys/kern/subr_bus.c:2734-2739): if (!rle->res) panic("resource_list_ release: resource entry is not busy") — unconditional.
3. Re-attach after fallback → KASSERT: pci_setup_msix (sys/bus/pci/pci.c:1644-1645) KASSERT(cfg->msix.msix_table_res == NULL && ... msix_pba_res == NULL, ("MSI-X has been setup yet")). The audit guest runs INVARIANTS (sys/config/X86_64_GENERIC:56) — proven live: the vtpci KASSERT panic in run.3.log fired on this very kernel.

Trigger analysis (why the panic could not be forced live)

• The per-cpu MSI-X pool (ioapic_irqmaps[cpu], sys/platform/pc64/apic/ ioapic_abi.c:1248-1310) and the per-cpu irq_rman regions (ioapic_abi.c:1200-1245) are derived from the same map, so a vector that ioapic_abi_msix_alloc hands out is by construction resolvable by rman_reserve_resource. Failure of the resource step therefore requires kernel-memory exhaustion inside rman/kmalloc or a genuine desync — neither forceable from QEMU device config or loader tunables.
• Pool exhaustion (EMSGSIZE from ioapic_abi_msix_alloc) fails earlier, in pci_alloc_msix_vector — the handled path (live-validated, see below).
• Manifestation 3 needs a detach+re-attach of the nvme driver; nvme is compiled into X86_64_GENERIC and no sys/modules/nvme exists, so no kldunload/kldload cycle is possible on this guest.

What WAS verified live (guest, INVARIANTS kernel)

Guest: DragonFly 6.5-DEVELOPMENT v6.5.0.1712.g89e6a-DEVELOPMENT (X86_64_GENERIC), QEMU/KVM.

1. Baseline (vm_nvme.sh: 1 controller msix_qsize=65, -smp 6): nvme0: mapped 7 MSIX IRQs — nirqs = min(65, ncpus+1) exactly per nvme_attach.c:153-156; nominal queue map; disk attached. (run.log)
2. Empty pool (hw.ioapic.msi_start=224 → pool [224,192)=∅): first pci_alloc_msix_vector fails at i=0, unwind no-op, legacy fallback: nvme0: mapped 1 normal-int IRQs, "no MSI-X support, limit comqs to 4", disk attached — no panic. (run.2.log)
3. Mid-loop failure (vm_nvme24.sh: 24 controllers, -smp 2, hw.ioapic.msi_start=160 → pool 2×32=64 vs 24×3=72 demanded): nvme0..nvme20 mapped 3 MSIX IRQs (63 slots); nvme21 fails at i≥1 — the unwind loop at nvme_attach.c:193-199 executes with a live allocated vector, releases it via bus_release_resource + pci_release_msix_vector, falls back to mapped 1 normal-int IRQs and attaches its disk; nvme22/23 fail at i=0 → fallback. No nvme panic anywhere; enumeration proceeds. (run.3.log)

This brackets the defect precisely: every failure the kernel itself reports is handled safely; the only unhandled failure is the unchecked NULL from bus_alloc_resource_any.

Incidental live discovery (NOT this finding, different driver)

During experiment 2/3 the boot panicked after all nvme devices attached:

panic: MSI-X is not setup yet
pci_release_msix_vector_method() ...
vtpci_intr_release() +0xe1
vtnet_attach() +0x909

i.e. sys/dev/virtual/virtio/virtio_pci.c vtpci_intr_release() calls pci_release_msix_vector() on an error path where MSI-X was never set up / already torn down (KASSERT at sys/bus/pci/pci.c:1588-1590). Reproduced on every boot with an exhausted/empty MSI pool (run.2.log, run.3.log). This is a real, live-reproduced boot-time panic in virtio_pci.c, outside this batch's two files — surfaced to the orchestrator for a dedicated audit spawn.

Recommended fix

Check the resource allocation and break with an error; tear MSI-X down when falling back after a successful setup. See fix.diff.

Fix validation

Not performed: the panic itself does not reproduce on the emulator (see trigger analysis), so there is no baseline behavior to diff a patched kernel against. fix_status: not_testable.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.