DF-2660 / verdict.json
{ "finding_id": "DF-2660", "status": "not_reproduced", "reproduced": false, "impact": "none", "confidence": "likely", "verdict": "NOT REPRODUCED live: the specific KKASSERT(chain == NULL) at synchro.c:758 was never observed on the stock INVARIANTS guest. Six attempts against real, census-verified 3-master same-clid clusters; four genuine race attempts (f/g/h/i) each drove a live 400-insert hammer2_sync_insert() backlog racing 6 full-speed rm/echo hammer loops (f: H2NOD_COUNT=3, BACKLOG_SEEN poll=0, 400 inserts in syslog) - and every attempt was killed first, 4/4 with an identical signature, by a sibling pre-existing NULL-deref on the same uncoordinated path: hammer2_inode_chain_and_parent.cold.9+0x15 (hammer2_inode.c:446-453 dereferences chain->parent after its own chain==NULL case) via hammer2_chain_inode_find+0x97 (hammer2_chain.c:5678) reached from the xop unlink/lookup paths (hammer2_xops.c:472/298/96) whenever a VOP touches a name whose in-core inode was just created by the sync threads dodefer machinery with a partially-populated cluster array. Shadowing analysis: the root scan inserts inode-key chains (all dodefer nips materialize) before the dirent-key chains (the KKASSERT surface), so from the start of the dirent phase every hammer rm of any synced name is primed for the sibling panic, while the assert needs a same-key VOP create inside one microsecond-scale lock handoff of the exact key being inserted - the sibling window is orders of magnitude wider and won 4/4. What IS empirically proven (4/4, dumps written): concurrent VOP file operations against a live sync catch-up pass on a 3-master rw cluster panic the stock kernel - the findings broader concurrency class, at the sibling site. The cited window itself remains structurally code-certain (inode unlocked at synchro.c:458, drop-locks/relock-EX/re-lookup/KKASSERT at 741-758).", "exploit_chain": "", "evidence": "VERDICT.md (shadowing analysis + choreography lessons); attempts_fghi.console.txt (attempt telemetry + all four sibling-crash backtraces, reconstructed from the session after the guest logs were lost to the fix-build snapshot revert); env.txt; fix.diff (EAGAIN replacement); fix_run.log (no-regression on kernel #1).", "kernel_refs": [ "sys/vfs/hammer2/hammer2_synchro.c:458", "sys/vfs/hammer2/hammer2_synchro.c:555", "sys/vfs/hammer2/hammer2_synchro.c:741", "sys/vfs/hammer2/hammer2_synchro.c:753", "sys/vfs/hammer2/hammer2_synchro.c:758", "sys/vfs/hammer2/hammer2_chain.c:113", "sys/vfs/hammer2/hammer2_inode.c:446", "sys/vfs/hammer2/hammer2_chain.c:5678", "sys/vfs/hammer2/hammer2_xops.c:472" ], "poc_changes": "No seed PoC existed (audit-time finding, status was untested). Built the race rig incrementally over attempts a-i: separate top-level merge mountpoints + h2nod census verification (mount rc 64 is ambiguous), vn attach guards, quiescent-entry-then-unleash choreography (VOP churn >=80 root-tid/s starves the sync entry ipcluster into 60s h2coll parks - the pass never starts), syslog-based backlog detection (kern.msgbuf saturates under debug floods), hash-order and reversed-order hammer sweeps (trailing sweeps never meet the insertion frontier).", "attempts": 6, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 7200, "guest_dirty": 0, "build_cmd": "no compile (shell rig); sh -n syntax check", "run_cmd": "guest root: sh /root/poc/df2660/trigger_df2660.sh <label> <vn-base>", "code_hash": "3c567c859dd3580fe629ee52404ada86370535f0c76b423ff3b1ab63a3a088d2", "notes": "Recommended follow-ups: (1) file the sibling defect (hammer2_inode_chain_and_parent NULL deref via hammer2_chain_inode_find, reproduced 5/5 sessions incl. the DF-2658 rig and once by mere background log rotation) - fixing it is the prerequisite for any live re-attempt of this findings assert; (2) re-run this trigger on a kernel carrying that NULL guard: with racing VOPs surviving, the reversed-sweep choreography should reach the synchro.c:758 handoff (expected-remaining odds per attempt estimated 10-50%). Guest left reverted to with-src snapshot, up.", "recommended_fix": "Replace KKASSERT(chain == NULL) in hammer2_sync_insert() with an EAGAIN return when the re-lookup finds a raced chain (the racing creator wrote all columns consistently in one transaction; the pass retries) - fix.diff, compiled into validation kernel #1.", "fix_status": "not_testable", "fix_baseline_reproduced": false, "fix_patched_reproduced": false, "fix_verdict": "fix.diff compiled clean into validation kernel #1 (both fixes, BUILD_INSTALL_DONE_RC0). Race-specific behavioral validation is not testable on this guest: the sibling chain_inode_find NULL-deref pre-empts every racing VOP before the synchro.c:758 site can be reached (4/4), and since the baseline itself never reproduced, patched-vs-baseline comparison has no signal. No-regression PASSED: the DF-2658 fixed-kernel run (../DF-2658/run_fixed.log) drove 400+ sync_insert/sync_replace calls through the patched hammer2_synchro.c on a real 3-master cluster to full convergence with no EAGAIN-related errors and no new panic sites.", "fix_evidence": "findings/poc/DF-2660/fix_run.log; findings/poc/DF-2658/run_fixed.log (same kernel, full catch-up pass); findings/poc/DF-2658/fix_build.log" } |