⬢ DragonFlyBSD Kernel Audit
← triage · dashboard
DF-4699

nvme(4): admin commands have no completion timeout — a stalled controller permanently wedges the admin thread and hangs detach/shutdown in nvwend

Summary

nvme_wait_request loops forever on SUBMITTED with only a 1hz re-sleep and every admin state function calls it holding admin_lk; a stalled controller wedges the admin thread, PROBED is never set, and stop_admin_thread sleeps forever - kldunload/detach/shutdown hang, deferred BIOs sleep forever. No watchdog fires. Bounded-wait + reset/failed-transition fix.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-4699 · 2 files
FileTypeDescriptionSize
README.md file 1.9 KB ↓ raw
verdict.json file 1.7 KB view raw
README.md file
↓ download raw

DF-4699 — nvme(4): admin commands have no timeout — a stalled controller permanently wedges the admin thread and all teardown

Where

sys/dev/disk/nvme/nvme.c:539-562 nvme_wait_request(); every admin-thread state function in sys/dev/disk/nvme/nvme_admin.c (nvme_admin_state_identify_ctlr:229-234, nvme_admin_state_make_queues:319-327/527-541, nvme_admin_state_identify_ns:594-630).

Trace

  • nvme_wait_request() (nvme.c:549-554) loops while (state == SUBMITTED) { poll; lksleep(req, lk, 0, "nvwait", hz); } — no deadline, no retry limit, no abort/reset escape. The 1 hz sleep timeout only re-loops.
  • All admin state functions call it while holding sc->admin_lk (taken once in nvme_admin_thread, nvme_admin.c:158).
  • A controller that accepts a submission and never posts the completion (broken/frozen firmware, malicious device — trivially done by a crafted emulator) wedges the admin thread forever.
  • Consequences: ADMIN_SIG_PROBED is never set, so nvme_stop_admin_thread() (nvme_admin.c:104-110) sleeps forever on admin_lk/PROBED → nvme_pci_detach never returns → kldunload/devd teardown/system shutdown hang in nvwend; the 1 hz loop in the thread also stops servicing ADMIN_SIG_REQUEUE, so deferred BIOs sleep forever (stalled I/O paths pile up). No watchdog fires (the thread sleeps legally).

Threat

DoS from a misbehaving controller (same trust boundary as DF-1813/DF-2512): host wedge at detach/shutdown, unkillable kernel thread + blocked module unload. No memory corruption.

PoC

Not drivable on the audit guest: QEMU 7.2's nvme device model always completes submitted admin commands; making it stall requires a patched emulator (none available on this host). untested (analysis-certain).

Fix sketch

Bound the wait (e.g. 30–60 s), then transition sc->admin_func = nvme_admin_state_failed, issue a controller reset (CC.EN toggle) or detach the device, and wake/abort waiters.

Fix verification

not_testable
baseline no→ patch + rebuild →patched clean

trigger cannot be generated by the available device model (QEMU 7.2 always completes admin commands)

per-fix-DF-4699

Confirmed kernel references

Detail

Evidence (decisive lines)

findings/poc/DF-4699/README.md; nvme.c:549-554, nvme_admin.c:104-110,158,229-234

PoC changes

no seed; analysis-only pack

Verified recommended fix

Bound admin waits with a deadline and fail over to nvme_admin_state_failed + controller reset instead of sleeping forever.

Verdict

Analysis-certain robustness/DoS defect: nvme_wait_request() has no deadline; every admin-thread state function blocks in it forever on a controller that never completes a command, holding admin_lk. PROBED is then never set, so nvme_stop_admin_thread() (and thus kldunload/device detach/system shutdown) hangs forever in nvwend, and the requeue service stops. Trigger requires a stalled/malicious controller; QEMU 7.2 always completes admin commands, so PoC left untested by design.