# DF-2632 — VERDICT

**Finding:** hammer2 flusher panics with `insert base %p overlapping
elements` (hammer2_base_insert, sys/vfs/hammer2/hammer2_chain.c:5308-5311)
when directory-entry keys are densely packed into one 64K dirhash
collision window — unprivileged local DoS.

**Classification: REPRODUCED (panic), fix FIXED.**

## 1. Reproduction (stock kernel #0, standalone, this run)

Fresh dedicated hammer2 image (512MB vn-over-tmpfs), one directory filled
with same-CRC32C names from `gen_names.c` (crc32c target 0xdeadbeef):

* `fill2632` batch=128: connection dies inside the first batch;
  `serial_panic_stock_standalone.log` (serial console):
  ```
  panic with -1 spinlocks held
  panic: td_critcount is/would-go negative! 0xfffff80118428f80 -1
  cpuid = 0
  crit_panic() at crit_panic+0x2f
  spin_unlock() at spin_unlock+0x49
  hammer2_base_insert() at hammer2_base_insert+0x652
  hammer2_chain_rename_obref() at hammer2_chain_rename_obref+0x66
  hammer2_chain_indirect_maintenance() at ...+0x411
  Debugger("panic")
  ```
* `denseprobe` batch=16 with sync+200 ms pause per batch
  (`density_probe.log`): last progress line `created=96`, panic before
  `created=112` → **the deterministic trigger threshold is 96 < N ≤ 112
  dense same-window dirents** (serial_panic_density96.log).
* (DF-2628's earlier transcripts, kept in this pack, show the same panic
  on a no-INVARIANTS rebuild with the structural message
  `panic: insert base 0xfffff8005753e000 overlapping elements at 0 elm ...`
  and the full `hammer2_flush_core → ... → base_insert` backtrace —
  proving it is not INVARIANTS-specific.)

## 2. Root cause (proven by the fix kernel's diagnostic output)

The fix kernel's deferred-insert diagnostic printed, on the first overlap:

```
hammer2: base_insert overlap deferred: parent 0xfffff8011943a100 type 2
    elm deadbeef60428000/7 type 2 vs base[0] deadbeef60428000/5 type 2
```

(type 2 = HAMMER2_BREF_TYPE_INDIRECT).  Decoded:

* The element being moved up by `hammer2_chain_rename_obref`
  (hammer2_chain.c:3501, called from indirect maintenance :4215) is an
  INDIRECT block with key `0xdeadbeef60428000`, **keybits 7** (range
  [0x...60428000, 0x...60428080)).
* The parent's base[0] already holds an INDIRECT with the **identical
  base key and keybits 5** (range [0x...60428000, 0x...60428020)) —
  a *nested* key range.

Mechanism: `hammer2_chain_create_indirect` normalizes an indirect's key
to its radix (`key &= ~((1<<keybits)-1)`, chain.c:3829) and the radix is
chosen by the split heuristic (`hammer2_chain_indkey_dir`, chain.c:4651)
each time an indirect must be created.  With a densely-packed window the
filler and the flusher interleave creates and collapses; two indirects at
different radix levels normalize to the SAME base key, so their ranges
nest instead of disjointly partitioning the space.  When the flusher later
collapses one (`hammer2_chain_indirect_maintenance`, chain.c:4065) and
moves its children into the parent (`rename_obref` → `base_insert`),
`hammer2_base_find` (:4921) returns index 0 for the moved element and the
overlap check at :5308-5311 fires:
`xkey (elm end) >= base[0].key` because the ranges share their base key.

Two additional defects compound at the panic site:

* The panic path at :5309 executes `hammer2_spin_unex(&parent->core.spin)`
  even though the indirect-maintenance call path does NOT hold that spin
  (the maintenance loop drops `chain->core.spin` around each move,
  chain.c:4191-4220).  The spurious unbalance drives `td_critcount`
  negative, which is what the INVARIANTS kernel actually reports first
  (`td_critcount is/would-go negative`, sys/sys/thread2.h:220-223).
* `panic()` then runs with corrupted crit/spin state ("panic with -1
  spinlocks held").

## 3. Trigger density / natural-name analysis (honest negative)

* Crafted names: the window needs only ~100 entries (96 < N ≤ 112 with
  per-16 syncs; DF-2628 saw 2-4K with coarser sync cadence — cadence
  changes how soon the flusher's maintenance runs against a dense window,
  not the mechanism).
* Natural names: `naturalprobe` replicates `hammer2_dirhash()` in
  userland.  For 25000 sequential + 25000 random 32-char names:
  `names=50000 windows=50000 max-per-window=1 windows-with->1=0` — the
  window index carries ~47 bits of CRC entropy, so natural names NEVER
  make a window dense (a 100-entry natural collision would need ~2^47
  names; even a single 2-entry collision has p ≈ 2^-47 per pair).
  Live: 50000 natural names in one hammer2 directory with sync every 512:
  **no panic** (`natural_probe.log`).  The finding therefore requires
  deliberate CRC32C multicollision construction — trivial with the MITM
  generator (5.5 s for 40000 names) but unreachable by accident.
  Severity Medium (local DoS) stands.

## 4. Fix (fix.diff, validated)

Three coordinated changes in `sys/vfs/hammer2/`:

1. `hammer2_base_insert()` returns `int`; the overlap case now skips the
   insert *before any stats side effects*, rate-limited diagnostic
   (krate 1/s) instead of `panic()`, and — critically — does NOT touch
   the spinlock (the old panic path's unconditional `spin_unex` was the
   critcount corruptor).  Callers updated: `hammer2_chain_rename_obref`
   defers by setting `HAMMER2_CHAIN_BLKMAPUPD|HAMMER2_CHAIN_UPDATE`
   (flush retries the blockref insertion later);
   `hammer2_flush_core` (hammer2_flush.c:1130) records the error, keeps
   UPDATE set and forces a flush retry.
2. `hammer2_chain_indirect_maintenance()` gains a **pre-flight check**
   BEFORE any destructive step: snapshot all live child key ranges
   (media blockrefs + RBTREE chains) of the indirect being collapsed and
   test them against every parent slot except the chain's own; if any
   range would overlap, abort the collapse with `return 0` (nothing has
   been modified, the flush defers, rate-limited notice).  This is the
   load-bearing fix: a mid-collapse deferral (iteration 1 of the fix)
   left skipped children in the collapsed chain and tripped
   `KKASSERT(chain->core.live_count == 0 && RB_EMPTY(...))` in
   `hammer2_chain_repchange` (chain.c:2314) — see fix_validation.log
   iteration note and serial console of kernel #1 in the run log; the
   pre-flight abort removes the deterministic trigger entirely and keeps
   the maintenance loop's invariants intact.
3. Rate-limited `krateprintf` diagnostics for both defer paths.

## 5. Fix validation (kernel #2 = X86_64_GENERIC + fix.diff)

A/B against stock (same KERNCONF, INVARIANTS on):

| test | stock #0 | fixed #2 |
|---|---|---|
| dense same-CRC fill | panic at 96-112 entries | 19818 entries, 0 errors, no panic (stopped at milestone; fill slows as the window deepens — deferred collapses retry — but completes error-free) |
| dense files after umount+remount | (n/a — panic) | 19818/19818 present |
| natural names 22k in one dir | ok | ok (max 1/window), umount clean |
| healthy image write/sync/umount/remount md5 | ok | ok (match) |

`fix_validation.log` + `fix_validation_part2.log`.

## 6. Notes / residual

* The deferred-collapse state leaves the (ghost-range) indirect in place;
  the flusher retries and re-defers at most 1/s per rate limiter.  No
  data loss observed (all 19818 files survived umount/remount).
* The underlying nested-range creation (two radix levels normalizing to
  the same key during create/maintenance interleaving) is a deeper hammer2
  topology issue; this fix makes it non-fatal and deferred, which is the
  correct minimal hardening for the panic.  Upstream may want to revisit
  `hammer2_chain_indkey_dir`'s split normalization.