# DF-2940 — VERDICT

**Finding**: `mpipe_done()` (sys/kern/kern_mpipe.c:122-159) tears the malloc
pipeline down on the strength of `KKASSERT(mpipe->free_count ==
mpipe->total_count)` with **no synchronization against in-flight users**.
Its only in-tree teardown caller is dm-crypt (`dm_target_crypt_destroy()` →
`dmtc_destroy_mpipe()`, sys/dev/disk/dm/crypt/dm_target_crypt.c:203-207,620),
whose bios are fully asynchronous: `dmstrategy()` drops the table reference
at dispatch (sys/dev/disk/dm/device-mapper.c:408-465) long before the crypto
completion path (`dmtc_bio_write_done` / `dmtc_bio_read_decrypt`) calls
`mpipe_free()`. Nothing in the teardown path (`dm_dev_remove_ioctl` /
suspend-load-`resume` table switch, dm_ioctl.c:480-524,521) waits for those
bios.

## Reproduced (2 full runs + 1 partial on stock INVARIANTS kernel #0/#1)

Guest: DragonFly 6.5-DEVELOPMENT #0/#1, X86_64_GENERIC, INVARIANTS on,
dm.ko + dm_target_crypt.ko + dm_target_delay.ko from /boot/kernel.

Recipe (`run.sh`, `run6.sh` style): vn(4) file-backed disk → dm-delay
(5-10 ms R/W) underlay `slow` → dm-crypt `vol` on top of `slow`;
24-40 parallel `dd` streams through `/dev/mapper/vol`; then repeated
`dmsetup suspend vol && dmsetup load vol <linear-on-vn0> && dmsetup resume vol`
(table switch destroys the crypt table with I/O in flight).

Kernel instrumentation (kprintf at `dm_target_crypt_destroy` /
`mpipe_done` entry, `dmopen`/`dmclose`, `dm_dev_remove_ioctl`) added in the
guest's /usr/src copy; stock semantics unchanged.

### Observations

1. **Contract violation at destroy entry — repeatedly**
   (`violations.txt`, deduped in `violations_summary.txt`):
   - `MPIDBG dmtc_destroy: rd free=20 total=20; wr free=8 total=20`  (12 checked out)
   - `MPIDBG dmtc_destroy: rd free=20 total=20; wr free=19 total=20` (1 checked out)
   Write mpipe buffers were checked out 1..12 at `dm_target_crypt_destroy`
   entry across dozens of reload cycles.

2. **Hard system wedge (DoS)**: within a few cycles of the same recipe the
   guest wedged twice: `dmsetup resume vol` stuck forever in kernel
   (wchan `waitmsg`) mid-destroy holding the table lock; 26-28 processes
   in permanent unkillable D-state on `dmtbl` (see `wedge_ps.txt`); kernel
   malloc pool exhausted (`dmesg` → "Cannot allocate memory"); ssh sessions
   die; only a power cycle recovers. In the wedge run the last console
   trace was `dmtc_destroy: wr free=12 total=20` — destroy entered with 8
   buffers outstanding and never returned.

3. **Stranded unkillable I/O**: 24+ dds left in D-state forever whose bios
   were parked on destroyed tables (mpipe retry queue orphaned / pdev
   teardown), the bio-leak face of the same race.

4. `dm_dev_remove_ioctl`'s `is_open` gate (dm_ioctl.c:354-359) blocks the
   plain remove path while any descriptor is open (and libdevmapper opens
   the device itself), so the reliable trigger is the **resume/reload
   path, which has no gate at all**.

### Why the KKASSERT did not fire on this guest

On INVARIANTS kernels the assert is *accidentally shielded* in the
single-device case: `dm_target_crypt_destroy` calls `dm_pdev_decr()`
(dm_target_crypt.c:618) **before** `dmtc_destroy_mpipe()`; when the pdev
refcount hits zero the `vrele`/VOP_CLOSE on the underlying device quiesces
its queue, so the delayed bios complete and the buffers return before
`mpipe_done` runs (observed: destroy-entry wr free=8 → mpipe_done
free=20/20). This is an incidental side effect, not synchronization: it
disappears whenever the pdev is shared (refcount > 0), and under load the
quiesce itself deadlocks (the wedge above). On production (non-INVARIANTS)
kernels the same sequence frees all cached mpipe buffers, `mpipe->array`,
`priv`, and exits the mpipe thread while in-flight bios still reference
them: the next `mpipe_free()` executes `mpipe->array[n] = buf` on freed
memory (UAF write into the freed `dm_target_crypt_config_t` slab object)
and `kfree()`s the buffer again; queued `struct mpipe_callback` retry
entries are leaked and their bios are never completed.

### Attack surface

`/dev/mapper/control` is root:operator 0640 (device-mapper.c:181) and
`dmioctl()` has **no priv_check** (device-mapper.c:241-280) — a member of
group `operator` (not root) can drive the whole reload/remove surface;
the unprivileged I/O itself comes through the block device. The wedge was
reproduced with nothing but `dmsetup suspend/load/resume` + `dd`.

## Fix validation

`fix.diff` part 2 (`dmtc_wait_mpipe_drain()` before `dm_pdev_decr`) was
built as a module in the guest and the exact wedge recipe re-run:
every one of the destroys after the fix logged
`MPIDBG dmtc_destroy(post-fix-wait): rd free=20 total=20; wr free=20 total=20`
(11 destroys, `boot.log`) — the mpipe contract violation is gone and the
reload loop kept making progress with the guest responsive (dmesg working,
ssh alive), where the baseline had wedged irrecoverably within ~2 cycles.

Under the extreme 40-dd flood a *later* stage of the teardown (the dm-core
`dm_pdev_decr` close-drain, reached only after the mpipe layer is now
clean) can still stall; that residual is the dm-framework
teardown-without-quiescing bug already on record as the DF-2443/2447/2453
family — it is outside kern_mpipe.c and outside this finding's scope.

`fix.diff` part 1 hardens `mpipe_done()` itself (bounded wait for checked-out
buffers, drain of queued retries before exiting the support thread) so the
file-level contract can no longer be violated silently; part 2 is the
caller-side fix that was runtime-validated.

## Bottom line

status: reproduced; impact: dos (kernel wedge + unkillable processes on
the INVARIANTS guest; UAF write + double-free + bio leak on production
builds per analysis above). Confidence: certain for the contract violation
and the wedge (instrumented, repeated); the production-kernel memory
corruption is derived from the same code paths but was not executed
(INVARIANTS guest asserts/threads first).