# DF-2660 VERDICT — NOT REPRODUCED (specific KKASSERT shadowed by a sibling # kernel panic in every live race attempt; race window itself code-certain) ## Bottom line Six race attempts were driven against real, verified 3-master hammer2 clusters on the stock INVARIANTS kernel (four of them — f, g, h, i — with a live catch-up backlog of 400 `hammer2_sync_insert()` calls racing 6 full-speed `rm fX; echo > fX` hammer loops). The specific `KKASSERT(chain == NULL)` at `sys/vfs/hammer2/hammer2_synchro.c:758` never fired, because every attempt was killed first — 4/4 with an identical signature — by a *sibling* pre-existing defect on the same uncoordinated concurrency path: Fatal trap 12: page fault, fault virtual address = 0x118 hammer2_inode_chain_and_parent.cold.9() at +0x15 (hammer2_inode.c:446-453) hammer2_chain_inode_find() at +0x97 (hammer2_chain.c:5678) -> from hammer2_xop_unlink/lookup/checkdirempty (hammer2_xops.c:472/298/96) `hammer2_inode_chain_and_parent()` dereferences `chain->parent` at hammer2_inode.c:453 without checking the `chain == NULL` case that its own lines 446-448 can produce. When a VOP (the hammer's `rm`) resolves a dirent whose in-core inode was just created by the sync thread's `dodefer` machinery (synchro.c:592-615) with a partially-populated cluster array, the xop worker NULL-derefs and panics the box. **Shadowing analysis (why the assert cannot win on stock):** the root-level scan inserts inode-key chains first (inum keys, bit 63 clear — all 200 `dodefer` nips materialize in the first half of the backlog) and dirent-key chains second (the KKASSERT surface). From the start of the dirent phase, *every* hammer `rm` of *any* synced name is primed to hit the sibling NULL-deref (nips exist for all names; each `rm`'s unlink resolves dirent->inode via `hammer2_chain_inode_find`), while the KKASSERT needs a same-key VOP create to land inside a single microsecond-scale lock handoff of the *exact* key the sync thread is inserting. The sibling window is orders of magnitude wider; empirically it won 4/4 attempts within seconds. What IS empirically proven on stock (4/4, dumps written): **concurrent VOP file operations against a live sync catch-up pass on a 3-master rw cluster panic the stock kernel** — the broader concurrency defect class the finding belongs to; just at the sibling site, not the cited assert. Additional gating mapped during the attempts (see attempts_fghi.console.txt): * Full-speed VOP churn (~80+ root-inode mtid changes/s) starves the sync thread's entry ipcluster xop (A/B feed mtid skew -> EINPROGRESS -> the 60s `h2coll` park, admin.c:990-998) — the pass never even starts. The working choreography: quiescent cluster until the pass starts, then unleash the hammer against the backlog (attempt f: `H2NOD_COUNT=3`, `BACKLOG_SEEN poll=0`, 400 inserts observed in syslog). * A hammer that trails the insertion frontier never collides on the exact key (attempt f/g/h: forward sweeps trail the faster sync thread; the frontier never crosses back). Attempt i used reversed-order sweeps to force a head-on frontier meeting — and died at the sibling NULL-deref at uptime 42s, as predicted by the shadowing analysis. * mount(8) returns 64 both for EBUSY-merges and for usage errors (hidden m2/m3 subdirs) — cluster formation must be verified via the `h2nod-