DF-2707 — duplicate transmitted DELETE: double RB_REMOVE + phantom rbtree

refdrop in kdmsg_state_cleanuptx() → state-tree corruption + premature

state free (UAF) — peer-driven, production (non-INVARIANTS) kernels

What this pack contains

• df2707_trigger.c — the trigger (modes: leak (DF-2708 evidence), wedge (DF-2710 evidence), attack (this finding)). It becomes the cluster peer of a kernel hammer2 iocom using the mount program's documented cluster_fd hook (hammer2_vfsops.c:1350-1357, same wire position as the userland hammer2 service daemon / a remote cluster node), then:
  1. parks the kernel write thread in fp_write() (small SO_SNDBUF on the kernel's socket end + junk SPAN transactions that each elicit an immediate 64-byte terminating reply);
  2. opens a victim LNK_SPAN transaction (stays open: msg_result);
  3. sends N=24 duplicate DELETE frames for the victim msgid while the writer is parked — on production kernels the rx-side assert (kern_dmsg.c:1076, DF-0018) is compiled out, each absorbed duplicate re-runs the callback and kdmsg_msg_reply() (its txcmd & DMSGF_DELETE check at kern_dmsg.c:2057 races the writer's cleanuptx) queues another terminating reply;
  4. drains → the writer transmits reply#1 (legit close: RB_REMOVE + "rbtree" refdrop), then reply#2 → kdmsg_state_msgtx() had reset txcmd (:1571), so kdmsg_state_cleanuptx() re-runs the close: RB_REMOVE on the already-removed node (stale-pointer writes into the live state tree) + a second "state on rbtree" refdrop (refcount underflow → premature kdmsg_state_free() while replies #3..#N still hold msg->state) → use-after-free;
  5. sprays CREATEs concurrent with the drain (reclaim of the freed slot) and churns the tree.
• run_df2707.sh — guest-side orchestration (image prep, newfs, build, run, dmesg capture, unmount).
• fix.diff — the verified fix (guard at the RB_REMOVE site keyed on the RBINSERTED invariant).
• logs (see VERDICT.md for interpretation):
• panic.invariants_baseline.txt — stock INVARIANTS kernel: same peer input panics at the rx-side assert kern_dmsg.c:1076 (DF-0018), proving the harness reaches the duplicate-DELETE machinery.
• panic.txt — non-INVARIANTS (production-equivalent) kernel: Fatal trap 12 page fault in kdmsg_state_tree_RB_REMOVE_COLOR — the double-remove corrupting the state tree. This is the finding.
• run.attack2_ssh.log / run.attack2_trigger.out — second attack run on the non-INVARIANTS kernel: no panic, but the iocom reader wedges after the drain (alternate manifestation).
• run.wedge_probe.txt — wedge mode: single-DELETE reader stall (DF-2710), separating the stall from the corruption.
• run.fix.log — same attack on the fix kernel: completes cleanly.

Build & run (from the repo root)

scp -F dfbsd-qemu/config findings/poc/DF-2707/df2707_trigger.c \
                      findings/poc/DF-2707/run_df2707.sh dfbsd:/root/poc/df2707/
dfbsd-qemu/vm.sh run_root 'sh /root/poc/df2707/run_df2707.sh attack 3 24'

The attack requires a kernel built without options INVARIANTS (production configuration); on the stock INVARIANTS kernel the same input panics earlier at kern_dmsg.c:1076 (DF-0018). Kernel build in guest:

sed -i.bak 's/^options \tINVARIANTS/#options \tINVARIANTS/' \
    /usr/src/sys/config/X86_64_GENERIC
cd /usr/src && make -j6 nativekernel KERNCONF=X86_64_GENERIC && \
    make installkernel KERNCONF=X86_64_GENERIC && reboot

Expected output

• baseline non-INVARIANTS: guest dies mid-run; serial console shows Fatal trap 12 ... Stopped at kdmsg_state_tree_RB_REMOVE_COLOR+0x6f (or an equivalent corruption crash / iocom wedge).
• fix kernel (fix.diff applied, still non-INVARIANTS): trigger prints all rounds + TRIGGER_DONE, UMOUNT_RC=0, no new kdmsg errors.

DF-2707 VERDICT — REPRODUCED (memory corruption, peer-driven, production kernels)

Bottom line

A cluster peer (wire position: the userland hammer2 service daemon's connection / any remote cluster node / the local cluster_fd mount hook, which is how the PoC drives it) can corrupt the kernel's dmsg state tracking on production (non-INVARIANTS) kernels by sending a handful of duplicate DELETE messages while the kernel's iocom write thread is blocked transmitting. The corruption is a double RB-tree unlink with stale pointers plus a phantom refcount drop, which frees the state early while further queued replies still reference it (use-after-free). Demonstrated as a kernel page fault inside kdmsg_state_tree_RB_REMOVE_COLOR on the non-INVARIANTS kernel; the identical input on the stock INVARIANTS kernel panics earlier at the rx-side assert (kern_dmsg.c:1076 = DF-0018), which is why this continuation was invisible to INVARIANTS testing.

Root cause chain (all in sys/kern/kern_dmsg.c unless noted)

1. Absorbed duplicate rx DELETE — kdmsg_state_msgrx() :1075-1077 only KKASSERTs against a second DELETE (rxcmd & DMSGF_DELETE already set); on non-INVARIANTS kernels the assert is a no-op (sys/sys/systm.h:93-118) and the duplicate is absorbed, then the callback runs again.
2. Re-reply race — the callback path (kdmsg_autorxmsg() :1195-1211 SPAN|DELETE → kdmsg_msg_reply(); same for CONN|DELETE :1161-1173) checks state->txcmd & DMSGF_DELETE (:2056-2058) without the iocom lock; txcmd's DELETE bit is only set by the write thread in kdmsg_state_cleanuptx() :1672 after transmission. While the writer is parked (blocked in fp_write() on a full socket — the peer simply stops reading), every absorbed duplicate queues another terminating reply.
3. Duplicate transmitted DELETE not guarded — kdmsg_state_msgtx() :1571 resets txcmd = cmd & ~DMSGF_DELETE for REPLY|CREATE|DELETE messages, so the KKASSERT((state->txcmd & DMSGF_DELETE) == 0) at :1671 passes even on INVARIANTS kernels for this route; kdmsg_state_cleanuptx() then re-runs the close: - RB_REMOVE() (:1678/:1683) on a node already removed → the tree.h macro re-walks the node's stale rbnode pointers and re-links former neighbors (sys/sys/tree.h:585-660) → live-tree corruption; - kdmsg_state_drop(state) "state on rbtree" (:1707) drops a ref that no longer exists → refcount underflow → premature kdmsg_state_free() while queued replies #3..#N still hold msg->state → UAF writes (state->txcmd |= DELETE, refs decrement, further stale RB_REMOVEs) into freed (and spray-reclaimable) kdmsg_state_t allocations.

Runs

Exploitability assessment (honest)

• Primitive: peer-controlled count (N replies) of (a) fixed-offset UAF writes into a freed 160-byte-class kdmsg_state_t (rbnode/flags/txcmd/refs fields), plus (b) repeated stale-pointer relinking of the live staterd_tree. The freed slot is same-zone reclaimable by peer-created states (every rx CREATE allocates from iocom->mmsg), so a later duplicate's RB_REMOVE can operate on a live replacement state's rbnode — compounding tree corruption with two owners of one node.
• Impact ceiling: remote kernel memory corruption → code execution on clustered HAMMER2 deployments (the peer position is exactly the trust boundary LNK_AUTH was supposed to guard and is unimplemented; receive-side CRCs are not verified). A full uid=0 chain was not constructed in this run: the attacker position is a cluster peer (not an unprivileged local user — the local cluster_fd route needs root to mount), and shaping reclaimed kdmsg_state_t content from the wire is limited to the msgid field (offset 120); everything else in the slot is kernel-chosen. The page-fault repro + the double-free/refcount-underflow derivation above establish the corruption primitive; weaponization would need a spray object with wire-controlled bytes in the 160-byte slab class and a target adjacent to the state trees, which the dmsg protocol surface does not directly offer. No "hard blocker" beyond that — the primitive is real and demonstrated.
• On INVARIANTS/debug kernels the same input is "only" a reachable assert (DF-0018) — production kernels are the vulnerable population.

Fix validation

fix.diff adds a guard at the only place the close sequence can legitimately run from (cleanuptx, before the RB_REMOVE): if the state is already fully closed (!(flags & KDMSG_STATE_RBINSERTED) inside rxcmd & DMSGF_DELETE), the duplicate DELETE transmit is discarded benignly (msg freed, temp ref dropped, return). The RBINSERTED flag is set at state creation (rx :904-916 / tx :1808-1817) and cleared exactly once by the single legitimate close (msgrx :1091 / cleanuptx :1686), so !RBINSERTED here ⇔ duplicate.

Rebuilt the non-INVARIANTS kernel with only this change, rebooted, re-ran the identical attack (attack 3 24): trigger completed all 3 rounds, printed TRIGGER_DONE, UMOUNT_RC=0, no kdmsg errors, no panic, no wedge (run.fix.log). Baseline (same kernel sans fix) panicked in RB_REMOVE_COLOR.

References

• sys/kern/kern_dmsg.c:1075-1077 (absorbed duplicate rx DELETE)
• sys/kern/kern_dmsg.c:2056-2058, 2101-2103, 2144-2146, 2184-2186 (unlocked txcmd re-reply checks)
• sys/kern/kern_dmsg.c:1195-1211, 1161-1173 (auto-reply on DELETE)
• sys/kern/kern_dmsg.c:1558-1573 (msgtx txcmd reset defeats :1671 assert)
• sys/kern/kern_dmsg.c:1670-1707 (unguarded close re-run; RB_REMOVE + refdrop)
• sys/sys/tree.h:585-660 (RB_REMOVE stale-pointer relink)
• sys/sys/systm.h:93-118 (KKASSERT INVARIANTS-only)
• hammer2_vfsops.c:1350-1357 (cluster_fd peer hook used by the PoC)