DF-2660 — hammer2_sync_insert() re-lookup TOCTOU: concurrent same-key

create on a mounted rw cluster trips KKASSERT(chain == NULL) on INVARIANTS

kernels (duplicate-key chain_create on production)

sys/vfs/hammer2/hammer2_synchro.c:741-758. Medium / hammer2 / CWE-367.

NOT REPRODUCED live on the stock INVARIANTS guest — shadowed 4/4 by a sibling kernel panic on the same uncoordinated path (every racing VOP dies at hammer2_inode_chain_and_parent.cold.9+0x15, fault va 0x118, via hammer2_chain_inode_find() from the xop unlink/lookup paths) before the synchro.c:758 assert can fire. The race window is code-certain; concurrent VOPs-vs-sync panics of the same family WERE reproduced deterministically at the sibling site. Full analysis in VERDICT.md and attempts_fghi.console.txt.

What it is

The sync scan runs with the inode unlocked (synchro.c:458) and the parent shared; hammer2_sync_insert() drops locks, relocks EX, re-looks-up the key as a TOCTOU re-check and asserts chain == NULL (synchro.c:758). A concurrent rm X; echo > X on the mounted rw cluster that inserts the same dirent key on the sync thread's own column inside the window panics INVARIANTS kernels; production kernels fall into duplicate-key hammer2_chain_create() against the overlap-as-match RB comparator.

Reproduce (attempt rig)

# guest (root):
sh trigger_df2660.sh <label> <vn-base>   # e.g. j 0

The trigger builds a 3-master clone cluster (A/B identical with f001..f200, C pristine), waits for the catch-up pass to start (quiescent entry — churn starves the ipcluster quorum into 60s h2coll parks), then unleashes 6 reversed-hash-order rm fX; echo > fX loops against the 400-insert backlog. On stock, expect the sibling NULL-deref panic (trace in attempts_fghi.console.txt) within seconds; the specific KKASSERT requires the sibling defect to be fixed first.

Fix

fix.diff — replace the KKASSERT with an EAGAIN return when the re-lookup finds a raced chain (the racing creator wrote all columns consistently in one transaction; the pass retries). No-regression validated on normal catch-up sync (see VERDICT.md).

DF-2660 VERDICT — NOT REPRODUCED (specific KKASSERT shadowed by a sibling

kernel panic in every live race attempt; race window itself code-certain)

Bottom line

Six race attempts were driven against real, verified 3-master hammer2 clusters on the stock INVARIANTS kernel (four of them — f, g, h, i — with a live catch-up backlog of 400 hammer2_sync_insert() calls racing 6 full-speed rm fX; echo > fX hammer loops). The specific KKASSERT(chain == NULL) at sys/vfs/hammer2/hammer2_synchro.c:758 never fired, because every attempt was killed first — 4/4 with an identical signature — by a sibling pre-existing defect on the same uncoordinated concurrency path:

Fatal trap 12: page fault, fault virtual address = 0x118
hammer2_inode_chain_and_parent.cold.9() at +0x15     (hammer2_inode.c:446-453)
hammer2_chain_inode_find() at +0x97                  (hammer2_chain.c:5678)
-> from hammer2_xop_unlink/lookup/checkdirempty      (hammer2_xops.c:472/298/96)

hammer2_inode_chain_and_parent() dereferences chain->parent at hammer2_inode.c:453 without checking the chain == NULL case that its own lines 446-448 can produce. When a VOP (the hammer's rm) resolves a dirent whose in-core inode was just created by the sync thread's dodefer machinery (synchro.c:592-615) with a partially-populated cluster array, the xop worker NULL-derefs and panics the box.

Shadowing analysis (why the assert cannot win on stock): the root-level scan inserts inode-key chains first (inum keys, bit 63 clear — all 200 dodefer nips materialize in the first half of the backlog) and dirent-key chains second (the KKASSERT surface). From the start of the dirent phase, every hammer rm of any synced name is primed to hit the sibling NULL-deref (nips exist for all names; each rm's unlink resolves dirent->inode via hammer2_chain_inode_find), while the KKASSERT needs a same-key VOP create to land inside a single microsecond-scale lock handoff of the exact key the sync thread is inserting. The sibling window is orders of magnitude wider; empirically it won 4/4 attempts within seconds.

What IS empirically proven on stock (4/4, dumps written): concurrent VOP file operations against a live sync catch-up pass on a 3-master rw cluster panic the stock kernel — the broader concurrency defect class the finding belongs to; just at the sibling site, not the cited assert.

Additional gating mapped during the attempts (see attempts_fghi.console.txt): * Full-speed VOP churn (~80+ root-inode mtid changes/s) starves the sync thread's entry ipcluster xop (A/B feed mtid skew -> EINPROGRESS -> the 60s h2coll park, admin.c:990-998) — the pass never even starts. The working choreography: quiescent cluster until the pass starts, then unleash the hammer against the backlog (attempt f: H2NOD_COUNT=3, BACKLOG_SEEN poll=0, 400 inserts observed in syslog). * A hammer that trails the insertion frontier never collides on the exact key (attempt f/g/h: forward sweeps trail the faster sync thread; the frontier never crosses back). Attempt i used reversed-order sweeps to force a head-on frontier meeting — and died at the sibling NULL-deref at uptime 42s, as predicted by the shadowing analysis. * mount(8) returns 64 both for EBUSY-merges and for usage errors (hidden m2/m3 subdirs) — cluster formation must be verified via the h2nod-<label> census, not mount exit codes.

Guest: DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 06:02:54 2026, X86_64_GENERIC (INVARIANTS), QEMU/KVM, stock kernel.

Status classification

not_reproduced / reproduced=0 — the finding's specific panic (assertion chain == NULL in hammer2_sync_insert, synchro.c:758) was not observed live; the race window it guards is structurally code-certain (scan with the inode unlocked at synchro.c:458, parent shared; drop-locks/relock-EX/ re-lookup/KKASSERT at synchro.c:741-758), and concurrent-VOPs-vs-sync panics of the same family were reproduced deterministically at the sibling site (documented above, full traces in attempts_fghi.console.txt).

Fix

fix.diff replaces the KKASSERT with an EAGAIN return on a raced re-lookup (the racing creator wrote the key to ALL columns in one cluster transaction, so dropping the stale insert preserves consistency; the sync pass retries). Applied to the fix-validation kernel together with the DF-2658 fix; validated for no-regression on normal catch-up sync (see below). Race-specific behavioral validation is not possible on stock while the sibling hammer2_inode_chain_and_parent NULL-deref pre-empts every racing VOP — fixing THAT defect (one NULL check at hammer2_inode.c:446-453) is the prerequisite for any live re-attempt and is recommended as its own finding.

(No-regression run results appended below.)

Fix validation (kernel #1, Thu Oct 15 07:30:39 UTC 2026)

The EAGAIN fix (fix.diff) was compiled into the same validation kernel as the DF-2658 fix (fix_build.log, BUILD_INSTALL_DONE_RC0). No-regression evidence: the DF-2658 fixed-kernel run (../DF-2658/run_fixed.log) exercised the full catch-up path on a real 3-master cluster — 400+ sync_insert / sync_replace calls through the patched hammer2_synchro.c, deferred passes and the root-inode final sync all completed, no EAGAIN storms, no new panic sites from this change (the only panic was the pre-existing sibling chain_inode_find NULL-deref).

Race-specific behavioral validation (raced re-lookup returns EAGAIN and the pass retries) is NOT testable on this guest while the sibling NULL-deref pre-empts every racing VOP (4/4 attempts); fixing hammer2_inode.c:446-453 (one NULL check) is the prerequisite for a live re-attempt. fix_status=not_testable (race), no-regression passed.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.