DF-5784 — hammer2 chain comparator key-range overflow (pass 3)

Finding (short)

hammer2_chain_cmp() (sys/vfs/hammer2/hammer2_chain.c:97-118) computes a chain's key range as end = key + ((hammer2_key_t)1 << keybits) - 1 with no validation that the range wraps the 64-bit keyspace. A crafted filesystem image can carry a blockref whose key + (1 << keybits) > 2^64 (e.g. key=0xC000000000000001, keybits=63 — both fully defined shifts, no UB). The in-memory chain for that bref then compares as "fully to the left" (end < other.key) of every other chain — including chains to its own left — so:

1. the chain is mis-placed in the parent's in-memory rbtree, and hammer2_chain_find()'s RB_SCAN (tree.h: comp < 0 descends RIGHT only) prunes it, so every lookup re-creates a duplicate chain from the same media bref (hammer2_chain_lookup → hammer2_chain_get → hammer2_chain_alloc, hammer2_chain.c:2610-2614), and
2. iteration keys regress: key_next = key + (1 << keybits) wraps below key_beg (hammer2_chain.c:2748-2749, 2594-2595), so directory iteration can never advance past the element — the readdir xop backend/frontend (hammer2_xop_readdir, hammer2_xop_collect's collect_key advance, hammer2_admin.c:1000-1019) live-lock.

Observed on the guest (stock INVARIANTS kernel): mounting the crafted image and issuing one directory read leaves the reading process stuck in getdents forever, unkillable (kill -9 → Operation not permitted), umount hangs, a CPU pegged with kernel interrupt-livelock messages (intr 16 on cpu1 at 40001/40000 hz, livelock limit engaged!), and the live chain counter vfs.hammer2.chain_allocs creeps upward (monotone leak; every lookup allocates a new duplicate chain that is dropped to 0 refs but remains in the rbtree until unmount — which can no longer happen). Threat model identical to DF-2616/DF-2617/DF-2618: crafted hammer2 image, mounted by root (or unprivileged user with vfs.usermount=1 and an owned vnode device).

Distinct from the known findings: DF-2618 is about overlapping ranges (cmp()==0 collisions/panics); DF-2606 is about UB shifts (keybits ≥ 64). This is a fully-defined-shift integer wrap of the range end that breaks the strict-weak ordering silently — no assert fires, no panic, the system just wedges and leaks.

Files

• forge_df5784.py — host-side forger (builds craftA/craftB from base.img)
• craftC variant: same patched key with keybits=0 (no wrap) — causality control, built with an inline python snippet (see build.sh)
• leakloop.c — drains a directory via readdir(3), prints live vfs.hammer2.chain_allocs
• run_df5784.sh — guest runner (control + crafted)
• run_wedge_transcript.txt — captured wedged-state evidence
• run_control_craftC.log — causality control output (200000 entries, CRAFTC_DELTA=0, umount OK)
• fix.diff — validated fix (skip insane media brefs in hammer2_base_find()'s forward scan + hammer2_bref_sane() helper)

Build

1. guest (root): sh mkbase.sh-equivalent (see build.sh) → base.img
2. host: python3 forge_df5784.py base.img craftA.img craftB.img
3. host: inline python for craftC.img (keybits=0 variant) — see build.sh
4. push to guest /root/poc/df5784/

Run (guest, root)

sh /root/poc/df5784/run_df5784.sh

Expected

• control (base.img): leakloop completes 1,000,000 entries quickly, chain_allocs delta ≈ 0, umount OK.
• crafted (craftB.img): leakloop never completes; the ssh session must be abandoned; from a second shell: ps shows the loop R2, kill -9 → Operation not permitted, umount hangs, chain_allocs creeps up.
• causality control (craftC.img, same key, keybits=0): 200,000 entries complete, CRAFTC_DELTA=0, umount rc=0 — proves the wrap (not merely the odd key) is the trigger.

Fix validation

fix.diff makes hammer2_base_find() skip media elements that fail hammer2_bref_sane() (unknown type or wrapped key range), exactly as it skips EMPTY slots — lookup/scan/readdir then never select the poisoned element. Validated by rebuilding the kernel in the guest and re-running: crafted image mounts, ls returns (entry invisible), leakloop completes with delta 0, umount OK, no wedge. See VERDICT.md.

DF-5784 — VERDICT

Status: reproduced (impact: dos — unkillable kernel-side live-lock + unbounded kernel memory growth from a mounted crafted hammer2 image; the mount cannot be unmounted afterwards). Fix validated (kernel rebuilt in guest: wedge gone, chain counter flat, umount clean).

Root cause (code)

hammer2_chain_cmp() (sys/vfs/hammer2/hammer2_chain.c:97-118) computes

c1_end = c1_beg + ((hammer2_key_t)1 << chain1->bref.keybits) - 1;   /* :109 */

Nothing between media ingest (hammer2_chain_get → hammer2_chain_alloc, hammer2_chain.c:2050-2098 / 177-237) and this comparator validates that key + (1 << keybits) does not exceed 2^64. A crafted image bref with key = 0xC000000000000001, keybits = 63 (all shifts fully defined) has

end = 0xC000000000000001 + 0x7FFFFFFFFFFFFFFF (mod 2^64) = 0x4000000000000000 < key

so cmp(wrapped, X) returns -1 ("fully to the left") for every X — and cmp(X, wrapped) also returns -1 whenever X.end < wrapped.key. Two consequences, both demonstrated on the guest:

1. rbtree mis-ordering + duplicate chains. The wrapped chain is inserted as a left child of everything (RB_INSERT uses cmp==0 only for true overlap — none here, so no assert fires even on INVARIANTS). hammer2_chain_find() (hammer2_chain.c:1931) runs RB_SCAN with hammer2_chain_find_cmp (hammer2_chain.c:1955-1969); DragonFly's RB_SCAN descends RIGHT-only when scancmp < 0 (sys/sys/tree.h, _##name##_RB_SCAN), so the scan prunes the mis-placed chain. hammer2_combined_find still selects its media bref via hammer2_base_find (hammer2_chain.c:4975-4986 forward scan breaks on scan->key > key_beg), so hammer2_chain_lookup (:2610-2614) calls hammer2_chain_get → a fresh duplicate chain is allocated, inserted, returned, dropped to 0 refs — and never found again. Live counter vfs.hammer2.chain_allocs creeps upward monotonically; the duplicates can only be reclaimed at unmount — which hangs (see 2).

2. Iteration regression + unkillable getdents live-lock. Iteration advance computes key_beg = key + (1 << keybits) (hammer2_chain.c:2748-2749 in hammer2_chain_next, :2594-2595 in hammer2_chain_lookup, and hammer2_base_find's scan_end :4992) — for the wrapped bref this is below the current position, so neither the chain iterator nor hammer2_xop_collect's collect_key = lokey+1 advance (hammer2_admin.c:1000-1019) can move the scan past the element. The readdir xop backend/frontend ping-pong (h2coll tsleep vs. FIFO stall wakeups) and the userspace getdents never returns.

Reproduction (stock INVARIANTS kernel #0 Jul 2 2026)

• control base.img: 1M-entry readdir drain completes instantly, chain delta 0, umount OK.
• craftB.img (f1 dirent bref key=0xC000000000000001 keybits=63; ancestors methods=CHECK_NONE; volhdr CRCs recomputed): single leakloop (readdir loop) never completes; process R2; kill -9 → Operation not permitted; umount → hangs (rc=124); serial console shows intr 16 on cpu1 ... livelock limit engaged!; chain_allocs creeps (818→830 over minutes). Transcript: run_wedge_transcript.txt.
• causality control craftC.img (SAME patched key, keybits=0 — no wrap): 200,000 entries complete in 66,666 passes, CRAFTC_DELTA=0, umount rc=0 (run_control_craftC.log). The wrap — not merely the odd key — is the trigger.

Fix validation

fix.diff: hammer2_bref_sane() helper (reject unknown blockref types and wrapped key ranges) + hammer2_base_find() forward scan skips insane media elements exactly like EMPTY slots. Applied to guest /usr/src, kernel #1 (Oct 12 15:27:54 2026) built+installed, rebooted:

• craftB on patched kernel: 1,000,000 entries complete (499,999 passes — the poisoned entry is simply not returned), CRAFTED_DELTA=0, umount OK (AFTER_UMOUNT=821). No wedge, no leak. (run_fix.log)
• DF-5785's craftA also validated on the same kernel (no panic).

Impact ceiling

DoS (wedge + resource exhaustion) — a memory-corruption escalation from this primitive was not identified: duplicates are properly allocated hammer2_chain objects; the rbtree remains structurally valid (RB_REMOVE is linkage-based, no comparator use); no OOB access occurs inside the loop. The system requires a reboot to recover the mount.

Not-a-duplicate note

DF-2618 = overlapping ranges (cmp==0 collisions → panic/tree-wipe); DF-2606 = keybits ≥ 64 UB shifts. This finding is a defined-behavior end-of-range wrap that silently breaks strict-weak ordering and iteration advancement — different trigger, different failure face (no panic at all, even on INVARIANTS).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.