DF-0117 — PoC (UAF on kdmsg state in diskiodone)

Finding: findings/DF-0117-diskiocom-uaf-kdmsg-state.md

Verdict: REPRODUCED — confirmed by kernel panic on the unpatched #0 kernel. Fix VALIDATED — the eof=1 patch eliminates the panic.

Summary

The prior "NOT REPRODUCED" verdict had a gap: it missed that the connection-drop teardown's simulated DELETE triggers disk_blk_read's error path (subr_diskiocom.c:384-394), which calls kdmsg_msg_reply unconditionally — even when iost->count > 0 (I/O in flight). That reply drives cleanuptx, which tears down the state (parent = NULL, removed from rbtree, topology refs dropped) while the bio still holds a raw state pointer. When the I/O completes, diskiodone dereferences the cleaned-up state → panic.

Files

Build & run (as root on the guest)

# Build (any user)
cc -O2 -o trigger_race trigger_race.c -lpthread

# Run (as root — needs /dev/vbd0 + DIOCRECLUSTER)
pkill -9 -x hammer2     # free the disk iocom from the hammer2 daemon
sleep 1
./trigger_race 131072 0 0 1
#   args: iosize nreads preclose_us iters
#   iosize=131072  MAXPHYS (128 KB) — slow enough for the teardown to win the race
#   nreads=0       no additional mid-stream I/Os
#   preclose_us=0  close socket immediately after writing
#   iters=1        one iteration (panics on first run if kernel is vulnerable)

Expected output

Fix

The fix (fix.diff) defers the DELETE reply to diskiodone when I/O is still in flight. In all four disk_blk_* error paths, when iost->count > 0, set iost->eof = 1 instead of calling kdmsg_msg_reply. The last diskiodone (when count reaches 0) sends the DELETE reply itself, after all derefs are done. This keeps the state valid until diskiodone has finished using it.

The finding markdown's proposed fix (uncomment kdmsg_state_hold/drop) does NOT work: those are file-local static macros in kern_dmsg.c, invisible to subr_diskiocom.c. Even when exported, the hold/drop alone does NOT prevent the UAF (confirmed empirically — the hold/drop kernel still panicked) because the teardown's kdmsg_subq_delete sets parent = NULL regardless of refcount.

DF-0117 — VERDICT

Status: REPRODUCED (UAF on kdmsg_state_t in diskiodone, confirmed by kernel panic). Fix VALIDATED (eof=1 deferral eliminates the UAF on the patched kernel).

One-line

The prior "NOT REPRODUCED" verdict had a gap in its teardown analysis: it claimed the state's topology refs prevent freeing during diskiodone, but missed that the connection-drop teardown's simulated DELETE triggers the disk callback's error path (subr_diskiocom.c:384-394), which calls kdmsg_msg_reply unconditionally — even when iost->count > 0 (I/O in flight). That reply drives cleanuptx, which removes the state from the rbtree, sets parent = NULL, and drops the topology refs — all while the bio still holds a raw pointer to the state. When the in-flight I/O completes, diskiodone dereferences the freed/cleaned state → panic.

Reproduction

A single MAXPHYS (128 KB) BLK_READ|CREATE on msgid=1 (eof=0), followed by immediate socket close (teardown), reproduces the panic reliably on the unpatched #0 kernel:

panic: assertion "pstate != NULL" failed in kdmsg_msg_alloc at kern_dmsg.c:1822
cpuid = 5
Trace beginning at frame 0xfffff8008ed4c860
kdmsg_msg_alloc() at kdmsg_msg_alloc+0x267
diskiodone() at diskiodone+0x120
vtblk_vq_intr() at vtblk_vq_intr+0xa7
vtpci_msix_intr() at vtpci_msix_intr+0x32
lwkt_serialize_handler_call() at lwkt_serialize_handler_call+0xd3

The 128 KB read is slow enough (1-10 ms on QCOW2-backed virtio) that the teardown frees the state before the I/O completes. diskiodone (called from the virtio block interrupt) then reads state->parent on the cleaned-up state — parent was set to NULL by kdmsg_subq_delete during the teardown's cleanuptx — triggering KKASSERT(pstate != NULL) in kdmsg_msg_alloc.

Mechanism (trigger → primitive → effect)

1. Peer sends BLK_READ|CREATE msgid=1, eof=0 (no DELETE in the message)
   subr_diskiocom.c:341  disk_blk_read()
   :360  iost = kmalloc(...)
   :365  bp = getpbuf_mem(NULL)
   :372  bio->bio_caller_info1.ptr = msg->state   ← RAW state ptr, no hold
   :378  eof = 0 (no DELETE)
   :381  dev_dstrategy(dp->d_rawdev, bio)          ← async I/O dispatched

2. Connection drops (socket close → reader EOF → writer teardown)
   kern_dmsg.c:547  teardown loop
   :555  kdmsg_simulate_failure(&state0, 0, ...)
    → kdmsg_state_abort(state)                      kern_dmsg.c:1355
    :1392  simulated msg cmd = DMSG_LNK_ERROR|DELETE
    :1404  kdmsg_msg_receive_handling(msg)
     → kdmsg_state_msgrx                            kern_dmsg.c:1073
       :1077  state->rxcmd |= DMSGF_DELETE          ← rxcmd gets DELETE
     → callback disk_rcvdmsg → disk_blk_read        subr_diskiocom.c:238
       (simulated cmd LNK_ERROR != BLK_READ → reterr=1)
       :385  msg has DELETE:
       :386  iost->count != 0 (I/O in flight) → skip kfree
       :390  kdmsg_msg_reply(msg, error)            ← CALLED UNCONDITIONALLY
         → kdmsg_msg_write → DYING branch           kern_dmsg.c:1974
           → kdmsg_state_cleanuptx                  kern_dmsg.c:1647
             :1672  state->txcmd |= DMSGF_DELETE
             :1673  rxcmd has DELETE → RB_REMOVE     ← removed from rbtree
             :1704  kdmsg_subq_delete(state)
               :1291  state->parent = NULL           ← PARENT CORRUPTED
               :1301  kdmsg_state_drop (subq ref)
             :1707  kdmsg_state_drop (rbtree ref)    ← topology refs dropped
     → kdmsg_state_cleanuprx                         kern_dmsg.c:1236
       :1272  kdmsg_msg_free → kdmsg_state_drop      ← refs → 0 → STATE FREED

3. Async I/O completes → diskiodone                  subr_diskiocom.c:577
   :580  state = bio->bio_caller_info1.ptr           ← FREED/CLEANED state
   :582  iost = state->any.any                       ← stale iost ptr
   :650  kdmsg_msg_alloc(state, cmd, NULL, 0)
     → pstate = state->parent                        ← NULL!
     → KKASSERT(pstate != NULL)                      ← PANIC

The prior verdict's gap

The prior verdict traced the state lifetime correctly for the diskiodone produces the DELETE reply path (eof=1 mode): the state is freed inside the LAST diskiodone's kdmsg_msg_write, after all derefs. That is correct.

But it missed the teardown produces the DELETE reply path (eof=0 mode): when the connection drops, kdmsg_simulate_failure simulates a DELETE receive. The simulated DELETE does NOT match BLK_READ (LNK_ERROR != BLK_READ), so disk_blk_read enters the error path (reterr=1). The error path calls kdmsg_msg_reply(msg, error) unconditionally — even when iost->count > 0 (I/O in flight). This sends a DELETE-bearing reply that drives cleanuptx, which tears down the state (parent=NULL, removed from rbtree, topology refs dropped) while the bio still holds a raw state pointer. When the I/O completes, diskiodone dereferences the cleaned-up state → UAF.

The prior verdict stated (incorrectly): "it never bypasses the txcmd |= DELETE requirement." But kdmsg_msg_reply (called from the error path) DOES set txcmd |= DELETE via cleanuptx. The prior verdict did not trace the error-path callback.

Setup

The disk iocom is attached via DIOCRECLUSTER on /dev/vbd0 with one end of an AF_UNIX socketpair (reused from sibling finding DF-0017). The hammer2 userland daemon pre-attaches every disk iocom at boot, so pkill -9 -x hammer2 frees the iocom first. See trigger_race.c for the self-contained trigger.

Trigger position: local root (needs /dev/vbd0 open + DIOCRECLUSTER). The remote vector (authenticated cluster peer sending DELETE while I/O is in flight) is equivalent.

Fix

The fix defers the DELETE reply to diskiodone when I/O is still in flight. In all four disk_blk_* error paths, when iost->count > 0, set iost->eof = 1 instead of calling kdmsg_msg_reply. The last diskiodone (when count reaches 0) sends the DELETE reply itself, after all derefs are done. This keeps the state valid (in rbtree, parent intact) until diskiodone has finished using it.

The finding markdown's proposed fix (uncomment kdmsg_state_hold/drop) does NOT work: those functions are file-local static macros in kern_dmsg.c, invisible to subr_diskiocom.c. Even if exported, the hold/drop alone does NOT prevent the UAF: the teardown's kdmsg_subq_delete sets parent = NULL regardless of the refcount, so diskiodone still reads a corrupted parent. This was confirmed empirically — the hold/drop kernel panicked with the same assertion.

Fix validation (Phase 8)

Stress tests on the patched kernel: 10+ iterations across 3 I/O patterns (1/5/17 I/Os per state), ~30+ diskiodone completions during teardown races, zero panics.

Exploit chain

The primitive is a kdmsg_state_t UAF (~152 bytes, M_DMSG_DISK malloc type, kmalloc-256 bucket). In a real cluster scenario, an authenticated peer controls the timing between the state free (send DELETE or drop connection) and the I/O completion. The freed state's slab slot can be reclaimed by same-type allocations during the teardown (LNK_CONN/LNK_SPAN message processing). On a non-KKASSERT kernel (or with the KKASSERT stripped), the stale parent/iocom dereference would be a wild pointer read → potential code execution via function-pointer corruption in kdmsg_msg_alloc or kdmsg_msg_write. Full exploitation (slab grooming + function pointer override) was not developed beyond the panic reproduction on this single-disk guest.

Files