DF-2940 — VERDICT

Finding: mpipe_done() (sys/kern/kern_mpipe.c:122-159) tears the malloc pipeline down on the strength of KKASSERT(mpipe->free_count == mpipe->total_count) with no synchronization against in-flight users. Its only in-tree teardown caller is dm-crypt (dm_target_crypt_destroy() → dmtc_destroy_mpipe(), sys/dev/disk/dm/crypt/dm_target_crypt.c:203-207,620), whose bios are fully asynchronous: dmstrategy() drops the table reference at dispatch (sys/dev/disk/dm/device-mapper.c:408-465) long before the crypto completion path (dmtc_bio_write_done / dmtc_bio_read_decrypt) calls mpipe_free(). Nothing in the teardown path (dm_dev_remove_ioctl / suspend-load-resume table switch, dm_ioctl.c:480-524,521) waits for those bios.

Reproduced (2 full runs + 1 partial on stock INVARIANTS kernel #0/#1)

Guest: DragonFly 6.5-DEVELOPMENT #0/#1, X86_64_GENERIC, INVARIANTS on, dm.ko + dm_target_crypt.ko + dm_target_delay.ko from /boot/kernel.

Recipe (run.sh, run6.sh style): vn(4) file-backed disk → dm-delay (5-10 ms R/W) underlay slow → dm-crypt vol on top of slow; 24-40 parallel dd streams through /dev/mapper/vol; then repeated dmsetup suspend vol && dmsetup load vol <linear-on-vn0> && dmsetup resume vol (table switch destroys the crypt table with I/O in flight).

Kernel instrumentation (kprintf at dm_target_crypt_destroy / mpipe_done entry, dmopen/dmclose, dm_dev_remove_ioctl) added in the guest's /usr/src copy; stock semantics unchanged.

Observations

1. Contract violation at destroy entry — repeatedly (violations.txt, deduped in violations_summary.txt): - MPIDBG dmtc_destroy: rd free=20 total=20; wr free=8 total=20 (12 checked out) - MPIDBG dmtc_destroy: rd free=20 total=20; wr free=19 total=20 (1 checked out) Write mpipe buffers were checked out 1..12 at dm_target_crypt_destroy entry across dozens of reload cycles.

2. Hard system wedge (DoS): within a few cycles of the same recipe the guest wedged twice: dmsetup resume vol stuck forever in kernel (wchan waitmsg) mid-destroy holding the table lock; 26-28 processes in permanent unkillable D-state on dmtbl (see wedge_ps.txt); kernel malloc pool exhausted (dmesg → "Cannot allocate memory"); ssh sessions die; only a power cycle recovers. In the wedge run the last console trace was dmtc_destroy: wr free=12 total=20 — destroy entered with 8 buffers outstanding and never returned.

3. Stranded unkillable I/O: 24+ dds left in D-state forever whose bios were parked on destroyed tables (mpipe retry queue orphaned / pdev teardown), the bio-leak face of the same race.

4. dm_dev_remove_ioctl's is_open gate (dm_ioctl.c:354-359) blocks the plain remove path while any descriptor is open (and libdevmapper opens the device itself), so the reliable trigger is the resume/reload path, which has no gate at all.

Why the KKASSERT did not fire on this guest

On INVARIANTS kernels the assert is accidentally shielded in the single-device case: dm_target_crypt_destroy calls dm_pdev_decr() (dm_target_crypt.c:618) before dmtc_destroy_mpipe(); when the pdev refcount hits zero the vrele/VOP_CLOSE on the underlying device quiesces its queue, so the delayed bios complete and the buffers return before mpipe_done runs (observed: destroy-entry wr free=8 → mpipe_done free=20/20). This is an incidental side effect, not synchronization: it disappears whenever the pdev is shared (refcount > 0), and under load the quiesce itself deadlocks (the wedge above). On production (non-INVARIANTS) kernels the same sequence frees all cached mpipe buffers, mpipe->array, priv, and exits the mpipe thread while in-flight bios still reference them: the next mpipe_free() executes mpipe->array[n] = buf on freed memory (UAF write into the freed dm_target_crypt_config_t slab object) and kfree()s the buffer again; queued struct mpipe_callback retry entries are leaked and their bios are never completed.

Attack surface

/dev/mapper/control is root:operator 0640 (device-mapper.c:181) and dmioctl() has no priv_check (device-mapper.c:241-280) — a member of group operator (not root) can drive the whole reload/remove surface; the unprivileged I/O itself comes through the block device. The wedge was reproduced with nothing but dmsetup suspend/load/resume + dd.

Fix validation

fix.diff part 2 (dmtc_wait_mpipe_drain() before dm_pdev_decr) was built as a module in the guest and the exact wedge recipe re-run: every one of the destroys after the fix logged MPIDBG dmtc_destroy(post-fix-wait): rd free=20 total=20; wr free=20 total=20 (11 destroys, boot.log) — the mpipe contract violation is gone and the reload loop kept making progress with the guest responsive (dmesg working, ssh alive), where the baseline had wedged irrecoverably within ~2 cycles.

Under the extreme 40-dd flood a later stage of the teardown (the dm-core dm_pdev_decr close-drain, reached only after the mpipe layer is now clean) can still stall; that residual is the dm-framework teardown-without-quiescing bug already on record as the DF-2443/2447/2453 family — it is outside kern_mpipe.c and outside this finding's scope.

fix.diff part 1 hardens mpipe_done() itself (bounded wait for checked-out buffers, drain of queued retries before exiting the support thread) so the file-level contract can no longer be violated silently; part 2 is the caller-side fix that was runtime-validated.

Bottom line

status: reproduced; impact: dos (kernel wedge + unkillable processes on the INVARIANTS guest; UAF write + double-free + bio leak on production builds per analysis above). Confidence: certain for the contract violation and the wedge (instrumented, repeated); the production-kernel memory corruption is derived from the same code paths but was not executed (INVARIANTS guest asserts/threads first).