DF-0017 — PoC (REPRODUCED)

Unbounded recursion in kdmsg_simulate_failure() / kdmsg_state_dying() (sys/kern/kern_dmsg.c:1346 / :1428) overflows the 16 KB LWKT kernel thread stack via a deep DMSG circuit-nesting chain. See VERDICT.md for the full analysis.

Status

REPRODUCED + FIX VALIDATED — kernel stack-overflow DoS (double-fault panic) on DragonFly master DEV 6.5-DEVELOPMENT #0. Prior inconclusive (could not obtain a connected DMSG iocom fd) is resolved: trigger.c performs the disk-open + socketpair + DIOCRECLUSTER setup itself. A single-fix kernel with DMSG_MAX_CIRCUIT_DEPTH=8 (fix.diff) eliminates the panic; the finding's proposed bound of 32 is empirically too high (33 levels still overflow — see VERDICT.md).

Files

• trigger.c — self-contained reproducer (the one to use). Opens /dev/vbd0, builds a socketpair, attaches one end to the kernel disk DMSG iocom via DIOCRECLUSTER, writes N chained CREATE messages (circuit nesting) + a root DELETE. Includes a drain thread.
• kdmsg_stackoverflow.c — the original wire-format builder (retained as the minimal trigger reference; it expects the caller to supply the fd).
• build.sh / run.sh — one-command build + run (run.sh also frees the disk iocom by killing the boot-time hammer2 daemon — see below).
• VERDICT.md — full narrative + evidence.
• build.log, run.log, panic.txt, env.txt — untrimmed logs.

Build (on the DragonFly guest, as root)

./build.sh        # cc -o trigger trigger.c -lpthread

Run (as root)

# (once, to capture the panic on a headless guest)
echo 'console="comconsole"' >> /boot/loader.conf && reboot

./run.sh          # default depth 300; frees the disk iocom, then fires
# or:  ./run.sh 300

Why run.sh kills the hammer2 daemon

On this image the userland hammer2 cluster daemon (pid "hammer2") connects every disk iocom at boot via DIOCRECLUSTER (sbin/hammer2/cmd_service.c:898) and relays peer DMSG traffic (TCP 987) into the kernel. That leaves each disk iocom's reader blocked in fp_read(), which deadlocks a follow-on DIOCRECLUSTER in kdmsg_iocom_reconnect() (kern_dmsg.c:141). Killing the daemon breaks the pipes, the readers exit, and a fresh DIOCRECLUSTER succeeds. The root fs (hammer2 on vbd0s1d) has its own kernel iocom and is unaffected. run.sh does pkill -9 -x hammer2 first.

Expected output

• Bug present: kernel panic — Fatal double fault (total stack exhaustion, page-aligned rsp), guest freezes in DDB. vm.sh reset to recover.
• Control (./run.sh 5): no panic, trigger exits 0 (shallow chain).
• Fixed kernel (depth cap): trigger exits 0, no panic at any depth.

Reachability / impact

• Local (DIOCRECLUSTER): needs to open a raw disk node (/dev/vbd0 is root:operator crw-r-----); unprivileged users are denied. → root/operator.
• Remote (HAMMER2 cluster relay, TCP 987): LNK_AUTH unimplemented, receive-side CRC not checked → a network peer/MITM can forge the chain. → unauthenticated DoS for HAMMER2-clustered deployments.
• No guard page on the LWKT stack → also an (uncontrolled) kernel memory-corruption primitive; realistically reliable impact = DoS.

DF-0017 — VERDICT

Status: REPRODUCED (kernel stack-overflow DoS; also an uncontrolled kernel memory-corruption primitive). Prior inconclusive resolved: the iocom-fd setup gap is solved and the bug fires on master DEV.

One-line

Unbounded recursion in kdmsg_simulate_failure() / kdmsg_state_dying() overflows the 16 KB LWKT kernel thread stack when a DMSG peer builds a deep circuit-nesting chain. A 300-deep chain deterministically panics the kernel with a double fault (total stack exhaustion); a 5-deep chain (control) does not. Reachable locally via DIOCRECLUSTER (root/operator) and remotely via the unauthenticated HAMMER2 cluster relay (TCP 987).

The bug, confirmed in source

struct kdmsg_state (sys/sys/dmsg.h:735) has a subq child list and no depth/nesting counter anywhere. Two routines walk that tree recursively with no depth bound:

• kdmsg_simulate_failure() — sys/kern/kern_dmsg.c:1321, recursive call at :1346: kdmsg_simulate_failure(substate, 1, error);
• kdmsg_state_dying() — sys/kern/kern_dmsg.c:1421, recursive call at :1428: kdmsg_state_dying(scan);

The deep chain is built on the receive path. The CREATE case (kern_dmsg.c:850 case DMSGF_CREATE:) selects a parent state pstate from the attacker-supplied msg->any.head.circuit (:868-879, RB_FIND by msgid in staterd_tree) and links the new state as its child at :917 TAILQ_INSERT_TAIL(&pstate->subq, state, entry). There is no depth check. So a peer sends CREATE #1 with circuit=0 (child of state0), CREATE #2 with circuit=1 (child of state #1), ... CREATE #N with circuit=N-1, building an N-deep linear chain.

The recursion is triggered by teardown:

• kdmsg_state_cleanuprx() — sys/kern/kern_dmsg.c:1236. When a state with a non-empty subq receives a DELETE (:1247), it calls kdmsg_simulate_failure(msg->state, 0, DMSG_ERR_LOSTLINK) at :1255.
• Write-thread teardown — sys/kern/kern_dmsg.c:547-555. On connection close it calls kdmsg_simulate_failure(&iocom->state0, 0, DMSG_ERR_LOSTLINK) at :555 for any leftover states.

Inside kdmsg_simulate_failure(state, meto=1, ...) each level also calls kdmsg_state_abort(state) (:1336) which calls kdmsg_state_dying(state) (:1368) — itself an unbounded recursive walk over the remaining chain. So the overflow is driven by both recursive functions.

The receive path does not verify the DMSG CRCs: kdmsg_iocom_thread_rd() (kern_dmsg.c:326-394) checks only the magic (:343) and sizes; hdr_crc/aux_crc are computed only on transmit (:2009-2012). So forged wire messages are accepted. LNK_AUTH is unimplemented (no kernel-layer auth on cluster links).

LWKT kernel thread stack = LWKT_THREAD_STACK = UPAGES*PAGE_SIZE = 4*4096 = 16384 bytes (sys/sys/thread.h:472, sys/cpu/x86_64/include/param.h:126). It has no guard page (kmem_alloc_stack in sys/vm/vm_extern.h:131-136 is just kmem_alloc1(..|KM_STACK) with no guard mapping), so the overflow corrupts adjacent kernel memory before double-faulting.

Setup (the prior blocker, solved)

A DMSG iocom fd is attached to the kernel disk-iocom parser by opening a raw disk device node and issuing DIOCRECLUSTER (sys/sys/diskslice.h:99, struct disk_ioc_recluster { int fd; }). The kernel does holdfp(curthread, recl->fd, -1) (subr_diskiocom.c:118) to obtain the struct file * and passes it to kdmsg_iocom_reconnect() (subr_diskiocom.c:141). The kernel reader thread then parses whatever is written to the other end of that fd as DMSG wire messages.

The PoC (trigger.c) is self-contained: it opens /dev/vbd0, builds an AF_UNIX SOCK_STREAM socketpair, issues DIOCRECLUSTER with one end, and writes the forged CREATE/DELETE messages to the other end (a drain thread absorbs kernel replies so the writer never blocks).

The reconnect-deadlock wrinkle. On this guest the userland hammer2 cluster daemon (pid 68, hammer2: hammer2 autoconn_thread, listens on TCP 987) connects every disk iocom at boot via DIOCRECLUSTER (sbin/hammer2/cmd_service.c:898), leaving each disk iocom's reader blocked in fp_read() on a pipe to the daemon. A follow-on DIOCRECLUSTER then deadlocks inside kdmsg_iocom_reconnect() (kern_dmsg.c:141, while (msgrd_td || msgwr_td)) because the stuck reader never wakes to notice KILLRX. Killing the daemon (pkill -9 -x hammer2) breaks the pipes, the readers get EOF and exit (msgrd_td -> NULL), and a fresh DIOCRECLUSTER then succeeds. The hammer2 root fs has its own kernel iocom (hmp->iocom) and is unaffected by killing the userland daemon — verified (root fs stays rw, ssh stays up). run.sh performs this kill as a documented setup step.

Evidence (decisive)

Run as root on DragonFly v6.5.0.1712.g89e6a-DEVELOPMENT (X86_64_GENERIC), kernel console switched to serial (console="comconsole") so the panic is captured in dfbsd-qemu/boot.log.

• Control — ./trigger 5: trigger exits 0, guest stays up, dmesg clean. Shallow chain, no overflow. (Proves the panic is depth-driven, not an artefact of the DIOCRECLUSTER setup.)
• Panic — ./trigger 300: guest freezes; serial console shows:

DOUBLE FAULT Fatal double fault rip = 0xffffffff806564d4 rsp = 0xfffff800ab38f000 (page-aligned == rbp: total stack exhaustion) panic: double fault dblfault_handler() at dblfault_handler+0x10c dblfault_handler() at dblfault_handler+0x10c Stopped at Debugger+0x7c: movb $0,0xbd77f9(%rip) db>

Reproduced twice (identical signature; only the exhausted-stack address differs). Full logs: run.log, panic.txt.

A double fault with a page-aligned rsp is the canonical signature of a kernel thread stack overflow on x86: the recursion exhausts the 16 KB stack, the stack pointer runs off the allocation, and the next push/fault finds no usable stack to dispatch even the page-fault handler -> double fault -> panic. The trace shows only dblfault_handler() because the original kdmsg frames are destroyed by the stack exhaustion (no recoverable frame to walk). There is no guard page, so the overflow is also a memory-corruption primitive that reliably manifests as a DoS.

Exploit chain

Not developed to root. The primitive is an uncontrolled kernel stack overflow into adjacent kernel memory (no guard page). Converting it to reliable code execution would require: (a) controlling the slab/heap layout adjacent to a chosen LWKT thread stack to place a victim object (function pointer / ucred *) at the overflow offset, and (b) surviving long enough past the overflow to dereference the corrupted object before the double-fault — the overflow happens inside a deep kernel recursion, so the double-fault lands almost immediately, making heap-grooming extremely fragile. The realistically reliable, defensible impact is the DoS (deterministic kernel panic). The original trigger PoC (kdmsg_stackoverflow.c) is retained as the minimal wire-format builder; trigger.c is the self-contained reproducer (setup + trigger).

Privilege / reachability note

• Local vector (DIOCRECLUSTER): requires opening a raw disk node (/dev/vbd0 etc.), which is root:operator crw-r-----. Unprivileged maxx (uid 1001, not in operator) is denied (Permission denied confirmed). So the local vector needs root or operator.
• Remote vector (HAMMER2 cluster relay): the hammer2 daemon listens on TCP 987 and relays peer DMSG traffic into the kernel disk iocom. LNK_AUTH is unimplemented and receive-side CRC is not checked, so a network peer (or MITM) can forge the chain-building CREATE messages. For HAMMER2-clustered deployments this is a remote, unauthenticated DoS — which is why the finding is rated High.

What changed vs the original PoC

The original kdmsg_stackoverflow.c only built the wire-format messages and expected the caller to supply a "connected DMSG fd" — which it never showed how to obtain, so it could not run (inconclusive). The new trigger.c is fully self-contained: it performs the disk open + socketpair + DIOCRECLUSTER setup itself, adds a drain thread to avoid reply-backpressure deadlock, and drives both trigger paths (root DELETE via cleanuprx, plus connection-close via the write-thread teardown). build.sh / run.sh make it one-command reproducible (run.sh also performs the documented hammer2-daemon kill needed to free the disk iocom for a local reconnect on this guest image).

Recommended fix

Cap circuit nesting depth at the source — the receive CREATE path — by adding a depth field to struct kdmsg_state and rejecting any CREATE whose parent is already at DMSG_MAX_CIRCUIT_DEPTH. This bounds the chain at the input, which in turn bounds the recursive teardown walks in kdmsg_simulate_failure() (kern_dmsg.c:1321) and kdmsg_state_dying() (kern_dmsg.c:1421) so they can no longer overflow the 16 KB LWKT thread stack. (Converting those two walks to iterative explicit-stack traversals remains a worthwhile defense-in-depth follow-up, but with the depth cap enforced they are bounded and safe.)

The verified fix lives in fix.diff and supersedes the finding markdown's proposed diff (which used a depth bound of 32 — see the critical correction below).

CRITICAL: the depth bound must be 8, NOT 32

The finding's ## Recommended fix and the first cut of this runner's fix.diff both proposed DMSG_MAX_CIRCUIT_DEPTH = 32. That value is too high and does NOT close the bug. Phase-8 fix validation (this run) proved it:

• A single-fix kernel built with the cap at 32 was driven by the same PoC (./trigger 300). The cap fired correctly — only 33 states (depths 0–32) formed the chain, CREATE #34 was rejected with the new circuit nesting too deep log, and #35–300 fell through to the pre-existing missing parent in stacked trans path. But the connection-close teardown STILL double-faulted (identical Fatal double fault / page-aligned rsp signature as the unpatched baseline).

• Root cause of the underestimate: the finding's "≈250 nesting levels suffice to overflow" assumed ~50–64 bytes per nesting level (≈1–2 frames). That is wrong because kdmsg_state_abort() re-enters the receive path (kern_dmsg.c:1404 kdmsg_msg_receive_handling(msg) → kdmsg_state_msgrx → kdmsg_state_cleanuprx → kdmsg_simulate_failure). So each nesting level is a ~5-function call cycle, not 1–2 frames, and the real per-level cost is ~485 B. 33 levels × ~485 B ≈ 16 KB → overflow, exactly as observed.

• The verified fix.diff therefore uses DMSG_MAX_CIRCUIT_DEPTH = 8 (max chain = 9 states, worst-case recursion ~9 levels ≈ 4.4 KB — well within the 16 KB stack with a >3× margin). Legitimate DMSG/HAMMER2 circuit nesting is 1–3, so 8 remains generously non-restrictive.

Fix validation (Phase 8) — VALIDATED

Built and booted a single-fix kernel (cap = 8) and re-ran the same PoC.

Before/after contrast (the decisive evidence):

BASELINE (#0, ./trigger 300):
  DOUBLE FAULT
  Fatal double fault
  rsp = 0xfffff800ab38f000     (page-aligned == rbp: total stack exhaustion)
  panic: double fault
  dblfault_handler() at dblfault_handler+0x10c
  db>                            (guest frozen in DDB)

PATCHED #1 (cap=8, ./trigger 300):
  [7] survived; no panic observed
  TRIGGER_EXIT=0
  boot.log: kdmsg: circuit nesting too deep (9), rejecting CREATE
  boot.log: kdmsg: missing parent in stacked trans   (x290, CREATEs #11-300)
  [double fault / panic count]: 0
  (guest responsive; depth 5000 also clean — cap bounds regardless of input)

The cap=8 fix is deterministic (4 consecutive deep runs, no panic) and correct (the cap fires at depth 9, bounding the chain to 9 states so the teardown recursion stays ~9 levels ≈ 4.4 KB, far under the 16 KB stack). The fix.diff is git apply-able (git apply --check rc=0) and the single-fix kernel compiled cleanly (NK_DONE rc=0).

Full logs: fix_build.log (full nativekernel output, rc=0), fix_run.log (patched-kernel PoC run + boot.log proof of no panic), panic_baseline.txt (the #0 double-fault signature).

DF-0017 — fix.diff validation notes

Authored by the runner post-verification against the read-only sys/ tree (the master DEV kernel that reproduced the panic), then built, booted, and validated in a single-fix kernel (Phase 8). Never applied to sys/.

Status: VALIDATED — fix closes the bug (cap = 8)

What the fix does

Caps DMSG circuit-nesting depth at the source — the receive CREATE path — so a peer can never build the arbitrarily deep parent→child chain whose teardown overflows the 16 KB LWKT thread stack. This is the root-cause fix; with the chain bounded, the recursive walks in kdmsg_simulate_failure() (kern_dmsg.c:1321) and kdmsg_state_dying() (kern_dmsg.c:1421) cannot overflow the stack, so their existing recursive structure becomes safe.

Changes

1. sys/sys/dmsg.h — add int depth; to struct kdmsg_state. state0 is covered because kdmsg_iocom_init() does bzero(iocom, sizeof(*iocom)) (kern_dmsg.c:113), so state0.depth = 0. Every other state is kmalloc(... M_ZERO), so the field defaults to 0.

2. sys/kern/kern_dmsg.c - #define DMSG_MAX_CIRCUIT_DEPTH 8 near the top (see "Why 8" below). - Receive CREATE path (kdmsg_state_msgrx, after pstate is resolved from msg->any.head.circuit): if pstate != state0 && pstate->depth >= DMSG_MAX_CIRCUIT_DEPTH, log circuit nesting too deep (%d) and break with error = EINVAL. The check sits before iocom->freerd_state is consumed (line 898), so a rejected CREATE returns cleanly through the done: path (error != 0 → skip state update at :1071, no state leak) and freerd_state remains available for reuse. - Set state->depth in the receive CREATE path (right after state->parent = pstate;) and in the transmit CREATE path of kdmsg_msg_alloc() (:1799) so a transmit-created state used as a circuit parent (via DMSGF_REVCIRC lookups in statewr_tree) carries an accurate depth and the receive-side cap cannot be bypassed.

Why depth = 8 (NOT 32) — empirically derived

The finding's ## Recommended fix and the first cut of this fix.diff both proposed DMSG_MAX_CIRCUIT_DEPTH = 32. Phase-8 validation proved 32 is too high — a single-fix kernel built with cap=32 was driven by the same PoC; the cap fired correctly (33 states, CREATE #34 rejected with circuit nesting too deep (33)), but the connection-close teardown STILL double-faulted with the identical stack-exhaustion signature.

Root cause of the underestimate: the finding assumed ~50–64 B per nesting level (≈1–2 frames). That is wrong because kdmsg_state_abort() re-enters the receive path — kdmsg_state_abort (kern_dmsg.c:1404) calls kdmsg_msg_receive_handling(msg) → kdmsg_state_msgrx → kdmsg_state_cleanuprx → kdmsg_simulate_failure. So each nesting level is a ~5-function call cycle, not 1–2 frames, and the real per-level cost is ~485 B. 33 levels × ~485 B ≈ 16 KB → overflow, exactly as observed.

DMSG_MAX_CIRCUIT_DEPTH = 8 gives a worst-case chain of 9 states (depths 0–8) → ~9-level recursion ≈ 4.4 KB, well within the 16 KB stack with a >3× margin. Legitimate DMSG/HAMMER2 circuit nesting is 1–3, so 8 remains generously non-restrictive.

Validation (Phase 8)

Scope note (defense-in-depth follow-up, NOT required to close the bug)

The finding's ## Recommended fix also suggests converting the two recursive subq walks to iterative (explicit-stack) traversals as defense-in-depth. With the depth cap enforced at 8, those walks are bounded (≤9 levels) and cannot overflow, so this fix does not rewrite them — converting kdmsg_simulate_failure in particular is non-trivial because its state->scan / goto again logic handles concurrent list mutation during traversal (kdmsg_state_abort can remove elements), and a faithful iterative rewrite risks introducing new bugs. The cap is the minimal, root-cause fix; an iterative rewrite remains a worthwhile hardening follow-up (and would let the cap be raised if a legitimate use case ever needs deeper nesting).

This runner-authored fix.diff supersedes the finding markdown's proposed diff (which used a bound of 32 — too high, empirically still overflows — and placed the cap after freerd_state consumption in one reading, leaking the freerd state on rejection; this version sits before the consumption and uses the empirically-safe bound of 8).