DF-5037 — VERDICT

Status: reproduced (impact: dos — permanent kernel deadlock of the whole dm subsystem; privileged local trigger). Fix: validated on a rebuilt kernel.

Root cause (static proof)

Two lock-acquisition orders exist in the dm subsystem and they invert:

Edge A — sys/dev/disk/dm/dm_pdev.c (the audited file's flaw). dm_pdev_insert() acquires the global dm_pdev_mutex EXCLUSIVE at dm_pdev.c:128 and holds it across blocking, dm-reentrant VFS operations: dm_dk_lookup() → vn_open() at dm_pdev.c:143/78, then VOP_GETATTR() (dm_pdev.c:288) and dev_dioctl(DIOCGPART) (dm_pdev.c:168). With dm-on-dm stacking (a table backed by /dev/mapper/X — explicitly supported; see the stacking comment at dm_pdev.c:92-99), vn_open runs dmopen() (device-mapper.c:200-217) → dm_dev_lookup() (dm_dev.c:105-118) which takes the global dm_dev_mutex EXCLUSIVE. Lock order: dm_pdev_mutex → dm_dev_mutex.

Edge B — sys/dev/disk/dm/dm_dev.c (partner edge). dm_dev_remove() (dm_dev.c:305-316) and dm_dev_remove_all() (dm_dev.c:318-345) hold dm_dev_mutex EXCLUSIVE across disable_dev(), which cv_waits for ref_cnt == 0 (dm_dev.c:73-75) while STILL HOLDING dm_dev_mutex, and (remove_all) across dm_dev_destroy() → dm_table_destroy() (dm_table.c:136-163) → target dtor (e.g. dm_target_linear.c:157) → dm_pdev_decr() (dm_pdev.c:242-259) which takes dm_pdev_mutex. Lock order: dm_dev_mutex → dm_pdev_mutex.

Cycle ⇒ permanent deadlock. Two concrete interleavings, both proven live:

• Variant 1 (mutex ↔ mutex): loader thread inside dm_pdev_insert's vn_open (holds dm_pdev_mutex, blocked in dm_dev_lookup on dm_dev_mutex) while remove_all/remove (holds dm_dev_mutex) reaches any target dtor → dm_pdev_decr → blocks on dm_pdev_mutex.
• Variant 2 (busy-ref ↔ mutex, the deterministic duel): reloader holds a dm_dev busy reference for the whole dm_table_load_ioctl (busy at dm_ioctl.c:711, unbusy at :800) and, inside it, blocks on dm_dev_mutex (its old-table destroy does dm_pdev_decr → dm_pdev_free → vn_close(/dev/mapper/X) → dmclose → dm_dev_lookup; its ctor does vn_open → dmopen → dm_dev_lookup). Concurrently remove of that same device sleeps in disable_dev's cv_wait holding dm_dev_mutex waiting for the reloader's busy reference, which the reloader can never drop.

Reproduction (what was run)

./dm_abba duel /dev/vbd0 90 as root on the stock guest (dfbsd-qemu/vm.sh reset with-src, kldload dm dm_target_linear):

• Setup: create pbase + reload pbase (linear over /dev/vbd0) + resume pbase (required: until dm_dev_resume_ioctl → dmsetdiskinfo initializes the disk media info, the disk layer rejects opens of the node with ENXIO and every stacked pdev open would fail), then create stk.
• P1 hammers reload stk with a linear table backed by /dev/mapper/pbase (~3.5k ioctls/s; each reload destroys the previous inactive table → vn_close → dmclose → dm_dev_mutex, and re-opens the pdev → vn_open → dmopen → dm_dev_mutex, all while holding the busy ref on stk).
• P2 hammers remove stk (2-5 ms pacing).
• Watchdog detects both counters frozen >6 s; ps-watcher and dmsetup ls canaries mirror everything to /dev/console so evidence survives the wedge.

Result (see run.log, ps_detail.txt): both workers froze in D state on the complementary locks — wchan dmdevlis (the dm_dev_mutex, lockinit'd "dmdevlist" at dm_dev.c:443) and wchan dm_dev (dmv->dev_mtx, lockinit'd "dmdev" at dm_dev.c:238) — for 42 consecutive 2-second snapshots (84 s of frozen evidence, after which the guest was reset). Both dmsetup canaries were simultaneously wedged behind dm_dev_mutex, proving subsystem-wide loss: every dm ioctl, and therefore every I/O to any dm device, hangs. The parent's own deadlock-detector canary (pid 913, child of the duel parent) wedged too. ssh stayed alive in this variant; in an earlier manifestation the wedge also blocked new ssh sessions (pty allocation through devfs), i.e. practical impact reaches beyond dm.

Why this took several attempts (honest engineering log)

The same race surface is littered with other defects that pre-empted the deadlock interleaving and are all already-filed findings — see panics_observed.txt:

1. kHz-rate table churn with failing reloads feeds the DF-4474 (dm_ioctl.c error-path prop leaks) leak into the per-malloc-type limit (ks_limit = RAM/10, kern_slaballoc.c:371-372) → panic: prop dictionary: malloc limit exceeded in ~16-100 s.
2. The remove-hammer trips the DF-2443/DF-2447 dm_dev_remove no-ref-held UAF family (devstat_remove_entry NULL deref) unless remove/recreate is serialized.
3. A's vn_open in its nlookup phase racing B's destroy_dev hits a devfs sysref_get-on-PUTAWAY assertion panic (reference_dev via v_associate_rdev).
4. A raw-ioctl reproducer must resume the backing dm device (ENXIO until dmsetdiskinfo runs) or every stacked pdev open fails — which silently converts the PoC into the leak-panic generator of (1).

None of these change the verdict: with the churn tamed, the duel froze the kernel deterministically on the first properly-configured run.

Fix (fix.diff) and validation

Two-sided, breaking both edges:

• dm_pdev.c: dm_pdev_insert no longer holds dm_pdev_mutex across vn_open/VOP_GETATTR/dev_dioctl. The lock is only taken for the list lookup and insertion; after opening the vnode the lock is re-taken and the list re-checked, a lost race closing the loser's vnode (lookup + ref under the lock, so no new references can appear once unlinked).
• dm_dev.c: disable_dev only unlinks; the reference-drain cv_wait moved to dm_dev_wait_refs() called WITHOUT dm locks; dm_dev_remove and dm_dev_remove_all wait and destroy outside dm_dev_mutex (remove_all collects victims on a private TAILQ preserving reverse creation order).

Validation on the guest: make nativekernel + installkernel with fix.diff applied (kernel #1: Sat Sep 26 08:14:53 UTC 2026), then the exact same duel:

• run 1 (90 s): counters raced (175,513 reloads / 9,257 removes in 50 s), no freeze, all processes exited cleanly;
• run 2 (60 s): same;
• functional sanity: dmsetup create s1 over vbd0, create s2 stacked over /dev/mapper/s1, dd write+read through /dev/mapper/s2 (OK), clean removes.

Baseline wedged within one 90 s run; patched kernel survived everything. See run_patched.log.

Bottom line

A root-triggerable (device-mapper ioctls on /dev/mapper/control, 0640 root:operator) permanent kernel deadlock: two syscalls (reload with a dm-on-dm backing concurrent with remove/remove_all) wedge the entire dm subsystem and any I/O to dm devices, requiring a reboot. Privilege bar keeps it at Medium (consistent with the other dm-family findings), but the loss is total and permanent once triggered.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.