DF-2632 — hammer2 flusher panics on dense dirent packing

insert base %p overlapping elements in hammer2_base_insert() (sys/vfs/hammer2/hammer2_chain.c:5308-5311), reached from the flusher's indirect-block maintenance: hammer2_flush_core → hammer2_chain_indirect_maintenance → hammer2_chain_rename_obref → hammer2_base_insert. Unprivileged local DoS: any user with write access to one hammer2 directory can panic the box with ~100 specially-crafted filenames.

Trigger

hammer2_dirhash() (hammer2_subr.c:178-229) maps a delimiter-free name to key = 0x8.. | (crc32c << 32) | ((crc32c^(crc32c<<16)) & 0xFFFF0000) | 0x8000. All names sharing one CRC32C hash to the SAME key; hammer2_dirent_create() packs them into consecutive slots X8000, X8001, … of one 64K collision window (HAMMER2_DIRHASH_LOMASK = 0x7FFF).

gen_names.c (from findings/poc/DF-2628) emits arbitrary numbers of 8-char names sharing a chosen CRC32C (meet-in-the-middle, ~5.5 s for 40000 names on the guest).

fill2632.c creates the files (one empty file per name) in one directory with periodic sync(), printing progress so the panic-time count is recoverable from captured stdout.

Reproduce

# on the stock guest (root):
cc -O2 -o gen_names gen_names.c
cc -O2 -o fill2632 fill2632.c
truncate -s 512M /tmp/h2632.img
vnconfig -c vn1 /tmp/h2632.img
newfs_hammer2 -L DATA /dev/vn1
mkdir -p /mnt/h2632 && mount_hammer2 /dev/vn1@DATA /mnt/h2632
./gen_names deadbeef 40000 /tmp/names.txt
mkdir /mnt/h2632/dense
./fill2632 /mnt/h2632/dense /tmp/names.txt 128
# -> panic within the first batch (see serial_panic_stock_standalone.log)
# with fine-grained batches (denseprobe, batch=16, sync+200ms):
#    panics between 96 and 112 dense dirents (density_probe.log)

Expected (stock): panic: td_critcount is/would-go negative! with trace crit_panic ← spin_unlock ← hammer2_base_insert ← hammer2_chain_rename_obref ← hammer2_chain_indirect_maintenance (INVARIANTS kernel; the panic path's hammer2_spin_unex at chain.c:5309 runs while the caller holds no spin, driving td_critcount negative). A no-INVARIANTS build panics with the structural message instead (serial_panic_noinv.log, from DF-2628).

Fix validation (kernel #2, X86_64_GENERIC + fix.diff)

validate_fix.sh: 19818 dense same-CRC names created with ZERO errors, no panic (stock panics at 96-112), clean unmount, all 19818 files present after remount. Natural-name directories (22000 names, max 1 per dirhash window) unaffected. Stock-image write/sync/umount/remount md5 roundtrip unaffected. Fill throughput degrades as the window gets deep (deferred collapses retry, rate-limited 1/s console notice) — no panic, no data loss.

Files

fill2632.c / denseprobe.c / naturalprobe.c   triggers/probes
run_df2632.sh / run_probe.sh / run_natural.sh / validate_fix.sh
serial_panic_stock_standalone.log   fresh standalone stock panic (this run)
serial_panic_density96.log          panic at 96<N<=112 (batch=16)
density_probe.log / natural_probe.log / stock_run.log
fix_validation.log / fix_validation_part2.log
serial_panic_invariants.log / serial_panic_noinv.log   (DF-2628 transcripts)
fix.diff (shared DF-2632+DF-2633), VERDICT.md, verdict.json, manifest.json

DF-2632 — VERDICT

Finding: hammer2 flusher panics with insert base %p overlapping elements (hammer2_base_insert, sys/vfs/hammer2/hammer2_chain.c:5308-5311) when directory-entry keys are densely packed into one 64K dirhash collision window — unprivileged local DoS.

Classification: REPRODUCED (panic), fix FIXED.

1. Reproduction (stock kernel #0, standalone, this run)

Fresh dedicated hammer2 image (512MB vn-over-tmpfs), one directory filled with same-CRC32C names from gen_names.c (crc32c target 0xdeadbeef):

• fill2632 batch=128: connection dies inside the first batch; serial_panic_stock_standalone.log (serial console): panic with -1 spinlocks held panic: td_critcount is/would-go negative! 0xfffff80118428f80 -1 cpuid = 0 crit_panic() at crit_panic+0x2f spin_unlock() at spin_unlock+0x49 hammer2_base_insert() at hammer2_base_insert+0x652 hammer2_chain_rename_obref() at hammer2_chain_rename_obref+0x66 hammer2_chain_indirect_maintenance() at ...+0x411 Debugger("panic")
• denseprobe batch=16 with sync+200 ms pause per batch (density_probe.log): last progress line created=96, panic before created=112 → the deterministic trigger threshold is 96 < N ≤ 112 dense same-window dirents (serial_panic_density96.log).
• (DF-2628's earlier transcripts, kept in this pack, show the same panic on a no-INVARIANTS rebuild with the structural message panic: insert base 0xfffff8005753e000 overlapping elements at 0 elm ... and the full hammer2_flush_core → ... → base_insert backtrace — proving it is not INVARIANTS-specific.)

2. Root cause (proven by the fix kernel's diagnostic output)

The fix kernel's deferred-insert diagnostic printed, on the first overlap:

hammer2: base_insert overlap deferred: parent 0xfffff8011943a100 type 2
    elm deadbeef60428000/7 type 2 vs base[0] deadbeef60428000/5 type 2

(type 2 = HAMMER2_BREF_TYPE_INDIRECT). Decoded:

• The element being moved up by hammer2_chain_rename_obref (hammer2_chain.c:3501, called from indirect maintenance :4215) is an INDIRECT block with key 0xdeadbeef60428000, keybits 7 (range [0x...60428000, 0x...60428080)).
• The parent's base[0] already holds an INDIRECT with the identical base key and keybits 5 (range [0x...60428000, 0x...60428020)) — a nested key range.

Mechanism: hammer2_chain_create_indirect normalizes an indirect's key to its radix (key &= ~((1<<keybits)-1), chain.c:3829) and the radix is chosen by the split heuristic (hammer2_chain_indkey_dir, chain.c:4651) each time an indirect must be created. With a densely-packed window the filler and the flusher interleave creates and collapses; two indirects at different radix levels normalize to the SAME base key, so their ranges nest instead of disjointly partitioning the space. When the flusher later collapses one (hammer2_chain_indirect_maintenance, chain.c:4065) and moves its children into the parent (rename_obref → base_insert), hammer2_base_find (:4921) returns index 0 for the moved element and the overlap check at :5308-5311 fires: xkey (elm end) >= base[0].key because the ranges share their base key.

Two additional defects compound at the panic site:

• The panic path at :5309 executes hammer2_spin_unex(&parent->core.spin) even though the indirect-maintenance call path does NOT hold that spin (the maintenance loop drops chain->core.spin around each move, chain.c:4191-4220). The spurious unbalance drives td_critcount negative, which is what the INVARIANTS kernel actually reports first (td_critcount is/would-go negative, sys/sys/thread2.h:220-223).
• panic() then runs with corrupted crit/spin state ("panic with -1 spinlocks held").

3. Trigger density / natural-name analysis (honest negative)

• Crafted names: the window needs only ~100 entries (96 < N ≤ 112 with per-16 syncs; DF-2628 saw 2-4K with coarser sync cadence — cadence changes how soon the flusher's maintenance runs against a dense window, not the mechanism).
• Natural names: naturalprobe replicates hammer2_dirhash() in userland. For 25000 sequential + 25000 random 32-char names: names=50000 windows=50000 max-per-window=1 windows-with->1=0 — the window index carries ~47 bits of CRC entropy, so natural names NEVER make a window dense (a 100-entry natural collision would need ~2^47 names; even a single 2-entry collision has p ≈ 2^-47 per pair). Live: 50000 natural names in one hammer2 directory with sync every 512: no panic (natural_probe.log). The finding therefore requires deliberate CRC32C multicollision construction — trivial with the MITM generator (5.5 s for 40000 names) but unreachable by accident. Severity Medium (local DoS) stands.

4. Fix (fix.diff, validated)

Three coordinated changes in sys/vfs/hammer2/:

1. hammer2_base_insert() returns int; the overlap case now skips the insert before any stats side effects, rate-limited diagnostic (krate 1/s) instead of panic(), and — critically — does NOT touch the spinlock (the old panic path's unconditional spin_unex was the critcount corruptor). Callers updated: hammer2_chain_rename_obref defers by setting HAMMER2_CHAIN_BLKMAPUPD|HAMMER2_CHAIN_UPDATE (flush retries the blockref insertion later); hammer2_flush_core (hammer2_flush.c:1130) records the error, keeps UPDATE set and forces a flush retry.
2. hammer2_chain_indirect_maintenance() gains a pre-flight check BEFORE any destructive step: snapshot all live child key ranges (media blockrefs + RBTREE chains) of the indirect being collapsed and test them against every parent slot except the chain's own; if any range would overlap, abort the collapse with return 0 (nothing has been modified, the flush defers, rate-limited notice). This is the load-bearing fix: a mid-collapse deferral (iteration 1 of the fix) left skipped children in the collapsed chain and tripped KKASSERT(chain->core.live_count == 0 && RB_EMPTY(...)) in hammer2_chain_repchange (chain.c:2314) — see fix_validation.log iteration note and serial console of kernel #1 in the run log; the pre-flight abort removes the deterministic trigger entirely and keeps the maintenance loop's invariants intact.
3. Rate-limited krateprintf diagnostics for both defer paths.

5. Fix validation (kernel #2 = X86_64_GENERIC + fix.diff)

A/B against stock (same KERNCONF, INVARIANTS on):

fix_validation.log + fix_validation_part2.log.

6. Notes / residual

• The deferred-collapse state leaves the (ghost-range) indirect in place; the flusher retries and re-defers at most 1/s per rate limiter. No data loss observed (all 19818 files survived umount/remount).
• The underlying nested-range creation (two radix levels normalizing to the same key during create/maintenance interleaving) is a deeper hammer2 topology issue; this fix makes it non-fatal and deferred, which is the correct minimal hardening for the panic. Upstream may want to revisit hammer2_chain_indkey_dir's split normalization.