# DF-5037 — VERDICT

**Status: reproduced** (impact: dos — permanent kernel deadlock of the whole dm
subsystem; privileged local trigger). **Fix: validated on a rebuilt kernel.**

## Root cause (static proof)

Two lock-acquisition orders exist in the dm subsystem and they invert:

**Edge A — `sys/dev/disk/dm/dm_pdev.c` (the audited file's flaw).**
`dm_pdev_insert()` acquires the *global* `dm_pdev_mutex` EXCLUSIVE at
dm_pdev.c:128 and holds it across blocking, dm-reentrant VFS operations:
`dm_dk_lookup()` → `vn_open()` at dm_pdev.c:143/78, then `VOP_GETATTR()`
(dm_pdev.c:288) and `dev_dioctl(DIOCGPART)` (dm_pdev.c:168). With dm-on-dm
stacking (a table backed by `/dev/mapper/X` — explicitly supported; see the
stacking comment at dm_pdev.c:92-99), `vn_open` runs `dmopen()`
(device-mapper.c:200-217) → `dm_dev_lookup()` (dm_dev.c:105-118) which takes
the *global* `dm_dev_mutex` EXCLUSIVE. Lock order: **dm_pdev_mutex → dm_dev_mutex**.

**Edge B — `sys/dev/disk/dm/dm_dev.c` (partner edge).**
`dm_dev_remove()` (dm_dev.c:305-316) and `dm_dev_remove_all()`
(dm_dev.c:318-345) hold `dm_dev_mutex` EXCLUSIVE across `disable_dev()`,
which cv_waits for `ref_cnt == 0` (dm_dev.c:73-75) while STILL HOLDING
`dm_dev_mutex`, and (remove_all) across `dm_dev_destroy()` →
`dm_table_destroy()` (dm_table.c:136-163) → target dtor (e.g.
dm_target_linear.c:157) → `dm_pdev_decr()` (dm_pdev.c:242-259) which takes
`dm_pdev_mutex`. Lock order: **dm_dev_mutex → dm_pdev_mutex**.

Cycle ⇒ permanent deadlock. Two concrete interleavings, both proven live:

* **Variant 1 (mutex ↔ mutex):** loader thread inside `dm_pdev_insert`'s
  `vn_open` (holds `dm_pdev_mutex`, blocked in `dm_dev_lookup` on
  `dm_dev_mutex`) while `remove_all`/`remove` (holds `dm_dev_mutex`) reaches
  any target dtor → `dm_pdev_decr` → blocks on `dm_pdev_mutex`.
* **Variant 2 (busy-ref ↔ mutex, the deterministic duel):** reloader holds a
  dm_dev busy reference for the whole `dm_table_load_ioctl`
  (busy at dm_ioctl.c:711, unbusy at :800) and, inside it, blocks on
  `dm_dev_mutex` (its old-table destroy does `dm_pdev_decr` → `dm_pdev_free`
  → `vn_close(/dev/mapper/X)` → `dmclose` → `dm_dev_lookup`; its ctor does
  `vn_open` → `dmopen` → `dm_dev_lookup`). Concurrently `remove` of that same
  device sleeps in `disable_dev`'s cv_wait **holding `dm_dev_mutex`** waiting
  for the reloader's busy reference, which the reloader can never drop.

## Reproduction (what was run)

`./dm_abba duel /dev/vbd0 90` as root on the stock guest
(`dfbsd-qemu/vm.sh reset with-src`, `kldload dm dm_target_linear`):

* Setup: `create pbase` + `reload pbase` (linear over /dev/vbd0) + **`resume
  pbase`** (required: until `dm_dev_resume_ioctl` → `dmsetdiskinfo` initializes
  the disk media info, the disk layer rejects opens of the node with ENXIO and
  every stacked pdev open would fail), then `create stk`.
* P1 hammers `reload stk` with a linear table backed by `/dev/mapper/pbase`
  (~3.5k ioctls/s; each reload destroys the previous inactive table →
  vn_close → dmclose → dm_dev_mutex, and re-opens the pdev → vn_open →
  dmopen → dm_dev_mutex, all while holding the busy ref on `stk`).
* P2 hammers `remove stk` (2-5 ms pacing).
* Watchdog detects both counters frozen >6 s; ps-watcher and `dmsetup ls`
  canaries mirror everything to /dev/console so evidence survives the wedge.

Result (see `run.log`, `ps_detail.txt`): both workers froze in D state on the
complementary locks — wchan `dmdevlis` (the dm_dev_mutex, lockinit'd
"dmdevlist" at dm_dev.c:443) and wchan `dm_dev` (dmv->dev_mtx, lockinit'd
"dmdev" at dm_dev.c:238) — for 42 consecutive 2-second snapshots (84 s of
frozen evidence, after which the guest was reset). Both dmsetup canaries were
simultaneously wedged behind `dm_dev_mutex`, proving subsystem-wide loss: every
dm ioctl, and therefore every I/O to any dm device, hangs. The parent's own
deadlock-detector canary (pid 913, child of the duel parent) wedged too.
ssh stayed alive in this variant; in an earlier manifestation the wedge also
blocked new ssh sessions (pty allocation through devfs), i.e. practical
impact reaches beyond dm.

## Why this took several attempts (honest engineering log)

The same race surface is littered with other defects that pre-empted the
deadlock interleaving and are all already-filed findings — see
`panics_observed.txt`:

1. kHz-rate table churn with failing reloads feeds the DF-4474
   (dm_ioctl.c error-path prop leaks) leak into the per-malloc-type limit
   (`ks_limit = RAM/10`, kern_slaballoc.c:371-372) → `panic: prop dictionary:
   malloc limit exceeded` in ~16-100 s.
2. The remove-hammer trips the DF-2443/DF-2447 dm_dev_remove no-ref-held UAF
   family (`devstat_remove_entry` NULL deref) unless remove/recreate is
   serialized.
3. A's vn_open in its nlookup phase racing B's destroy_dev hits a devfs
   sysref_get-on-PUTAWAY assertion panic (reference_dev via v_associate_rdev).
4. A raw-ioctl reproducer must `resume` the backing dm device (ENXIO until
   dmsetdiskinfo runs) or every stacked pdev open fails — which silently
   converts the PoC into the leak-panic generator of (1).

None of these change the verdict: with the churn tamed, the duel froze the
kernel deterministically on the first properly-configured run.

## Fix (`fix.diff`) and validation

Two-sided, breaking both edges:

* **dm_pdev.c:** `dm_pdev_insert` no longer holds `dm_pdev_mutex` across
  `vn_open`/`VOP_GETATTR`/`dev_dioctl`. The lock is only taken for the list
  lookup and insertion; after opening the vnode the lock is re-taken and the
  list re-checked, a lost race closing the loser's vnode (lookup + ref under
  the lock, so no new references can appear once unlinked).
* **dm_dev.c:** `disable_dev` only unlinks; the reference-drain cv_wait moved
  to `dm_dev_wait_refs()` called WITHOUT dm locks; `dm_dev_remove` and
  `dm_dev_remove_all` wait and destroy outside `dm_dev_mutex` (remove_all
  collects victims on a private TAILQ preserving reverse creation order).

Validation on the guest: `make nativekernel` + `installkernel` with fix.diff
applied (kernel `#1: Sat Sep 26 08:14:53 UTC 2026`), then the exact same duel:

* run 1 (90 s): counters raced (175,513 reloads / 9,257 removes in 50 s), no
  freeze, all processes exited cleanly;
* run 2 (60 s): same;
* functional sanity: `dmsetup create s1` over vbd0, `create s2` stacked over
  `/dev/mapper/s1`, dd write+read through `/dev/mapper/s2` (OK), clean removes.

Baseline wedged within one 90 s run; patched kernel survived everything. See
`run_patched.log`.

## Bottom line

A root-triggerable (device-mapper ioctls on /dev/mapper/control, 0640
root:operator) permanent kernel deadlock: two syscalls (`reload` with a
dm-on-dm backing concurrent with `remove`/`remove_all`) wedge the entire dm
subsystem and any I/O to dm devices, requiring a reboot. Privilege bar keeps
it at Medium (consistent with the other dm-family findings), but the loss is
total and permanent once triggered.

## Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest `make -j6 nativekernel KERNCONF=X86_64_GENERIC` completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.