PoC DF-4528 — nvme(4) admin submission-queue lock-protocol violation

What this is

nvme_submit_request() documents its contract as "Caller must hold the queue lock" (nvme.c:509-513) and mutates shared submission-ring state without any internal lock:

cmd = &queue->ksubq[queue->subq_tail];       /* nvme.c:520 */
--queue->unsubmitted;                        /* nvme.c:521 */
if (++queue->subq_tail == queue->nqe) ...    /* nvme.c:522 */

The I/O path honors the contract (nvme_strategy_core() takes subq->lk around the submit, nvme_disk.c:336-338), but every admin-queue submitter violates it — nvme_getlog_ioctl() (nvme_ioctl.c:56), nvme_create_subqueue() (nvme.c:800), nvme_create_comqueue() (nvme.c:843), nvme_delete_subqueue() (nvme.c:877), nvme_delete_comqueue() (nvme.c:902) and all the admin thread's identify flows (nvme_admin.c:233, 319, 534, 598, 629) call nvme_submit_request() with no lock held. queue->unsubmitted is incremented under queue->lk in nvme_get_request() (nvme.c:480) but decremented unlocked here, so concurrent submitters corrupt the ring accounting, can double-write ksubq[subq_tail] (lost commands) and can drive nvme_get_request() to its NULL path (KKASSERT is a no-op on production, non-INVARIANTS kernels — sys/systm.h:116-118) or into the ring-full check with a corrupted counter.

Two concurrent submitters exist whenever the admin thread is issuing its attach/detach command sequence (identify controller, SET FEATURES number of queues, identify namespace × N, create/delete queue × N) while a user holds an open namespace fd and issues NVMEIOCGETLOG — e.g. hot-plug / rescan of a second... on the SAME controller: attach storm right after the namespace device appears, or the delete-queue storm during removal.

What was verified on the guest

• The admin ioctl path itself is live and was stress-tested clean: 1 × 60 s solo: 783,730 ioctls, 0 errors, max latency 6 ms; 6 × 90 s concurrent: ~1.35 M ioctls, 0 errors, max latency 0.82 s (ioctl_lk serializes the getlog submitters among themselves — see run.log). The race needs the admin thread as the second submitter.
• The deterministic trigger environment was built (nvmebang test module: on-demand driver attach/detach method invocation → repeated admin command storms); during these runs the teardown-race panic of DF-4529 (same files, same rig) fires first — the detach-side defect masks the submit-side race before attach-storm data could be collected.
• The identical locking violation also underlies the DF-4529 teardown race, which was reproduced to a panic on this rig.

Verdict: not_reproduced (dynamically), code-certain (statically) — the contract violation and its state mutations are unambiguous in the source; a clean dynamic demonstration requires the DF-4529 fix (or instrumented kernels) so the teardown panic does not preempt it.

Build / run

Same environment as DF-4529 (see ../DF-4529/README.md). The load tools:

cc -O2 -o /root/nvspam nvspam.c     # glob /dev/nvme*, loop NVMEIOCGETLOG
cc -O2 -o /root/nvslow nvslow.c     # per-call latency + errno stats
cc -O2 -o /root/nvgetlog nvgetlog.c # single-call sanity check

/root/nvgetlog once /dev/nvme0                    # sanity
/root/nvslow /dev/nvme0 60 0.1                    # solo load
for i in 1 2 3 4 5 6; do /root/nvslow /dev/nvme0 90 0.2 >/root/slow$i.log 2>&1 & done
# attach-storm window (needs DF-4529 fix in place to observe cleanly):
sysctl hw.nvmebang.action=2   # re-runs identify/make-queues admin storm

Fix

see fix.diff — admin submissions take the queue lock (new nvme_submit_admin_request() wrapper; all qid-0 submitters converted).