DF-2657 / verdict.json
{ "finding_id": "DF-2657", "status": "reproduced", "reproduced": 1, "impact": "dos", "confidence": "certain", "verdict": "Verified end-to-end. On stock, every sync thread of a same-clid 2-master cluster (and of master+slave-with-idx-master clusters, and of 1-element non-MASTER pmppfs like the boot LOCAL PFS) parks permanently in hammer2_xop_collect's EINPROGRESS loop ('h2coll') because hammer2_cluster_check computes quorum over all pfs_nmasters masters while the sync thread's xop was issued with hammer2_xop_start_except() excluding its own index - the excluded element is counted as an eternally in-progress master (umasters), so nmasters+umasters >= nquorum forever. umount -f then stalls ~65s in h2twait and fails rc=1 leaving the 4 sync threads parked forever, frozen xop cohorts, and hammer2_mntlk held (third mount D-state). Root-cause fix (participant-aware quorum via chk_mask + forward-only want_update gate) rebuilt in-guest: sync threads park in h2idle (0x h2coll), umount -f completes RC=0 in 0s, third mount instant, 6x single-device churn control clean, diverged rw cluster shows no regression of the fresher master and clean teardown. Residual: post-cluster-teardown mount churn trips the pre-existing DF-2620-family UAF at hammer2_primary_xops_thread+0x2d9 (identical trap site to DF-2631's independent fix builds; no-cluster control clean) - documented, not introduced by this fix.", "exploit_chain": "unprivileged-induced admin action (or any real >=2-master cluster): mount same-clid dd clones -> EBUSY merge -> all h2nod sync threads park in h2coll forever -> umount -f wedges/fails with threads stranded -> hammer2_mntlk held -> all later hammer2 mounts D-state -> shutdown degradation; recovery reboot-only. Not a privilege escalation: local DoS of the mount/teardown machinery.", "evidence": "run.log (stock: 4x h2coll pre-umount census, umount -f D4 h2twait +65s rc=1, third mount D3 mntlk); fix_run.log (kernel #1: A_H2COLL_COUNT=0/4x h2idle, A_UMOUNTF_RC=0 A_ELAPSED=0s, A_MOUNT3_RC=0, 6/6 solo churn clean, Phase B no-regression); fix.diff (10 hunks clean); fix_build.log.tgz (5.4MB BUILD_INSTALL_OK); panic_serial.log (4x DF-2620-family +0x2d9 traps on fix kernel, control clean); VERDICT.md (full narrative + open follow-ups)", "kernel_refs": [ "sys/vfs/hammer2/hammer2_synchro.c:411", "sys/vfs/hammer2/hammer2_synchro.c:454", "sys/vfs/hammer2/hammer2_synchro.c:672", "sys/vfs/hammer2/hammer2_synchro.c:421", "sys/vfs/hammer2/hammer2_cluster.c:348", "sys/vfs/hammer2/hammer2_cluster.c:462", "sys/vfs/hammer2/hammer2_cluster.c:522", "sys/vfs/hammer2/hammer2_admin.c:479", "sys/vfs/hammer2/hammer2_admin.c:988", "sys/vfs/hammer2/hammer2_admin.c:996" ], "poc_changes": "Wrote fresh trigger (DF-2631 methodology adapted: fresh newfs base image since the guest snapshot was clean, vn0-based newfs, added the decisive pre-umount h2coll census and +65s post-cycle census); wrote fix_validate.sh in three phases (wedge regression / diverged-cluster / stability) and had to run phases as discrete ssh commands after panic #1 froze the in-guest log writer; fix.diff authored against pristine sys/ after code-reading cluster_check/admin/synchro.", "attempts": 4, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC (baseline); fix kernel #1: Sun Aug 30 04:30:11 UTC 2026", "runtime_sec": 4200, "guest_dirty": 0, "build_cmd": "cd /usr/src && patch -p1 < /root/fix2657.diff && make nativekernel && make installkernel", "run_cmd": "sh /root/poc_df2657_trigger.sh # stock baseline; then on kernel #1: sh /root/poc_df2657_validate.sh (phases run discretely)", "code_hash": "57749ee19d6613873f23c038cfc8ae9238bac944452f261fad55a3bda0aaaa63", "notes": "fix_status=fixed for the filed defect (wedge family root cause); fix_kernel_uname=#1 Sun Aug 30 04:30:11; fix_baseline_reproduced=1; fix_patched_reproduced=0 (wedge gone). Residual DF-2620-family UAF exposure documented in panic_serial.log - same +0x2d9 site as DF-2631's builds, no-cluster control clean. Open follow-up: stale-column pull-in not observed on media (flush/trans-flag suspicion; diagnostics blocked by the UAF). IDs DF-2658..2660 filed from the same audit: sync_replace resize no-op OOB read, inode_chain_and_parent NULL-deref pair, sync_insert relookup KKASSERT race.", "recommended_fix": "Participant-aware quorum: thread the xop participant mask (chk_mask) from hammer2_xop_start_except() through hammer2_xop_collect() into hammer2_cluster_check() and compute nquorum over participating masters only; pair with a forward-only want_update gate (mtid < sync_tid) in hammer2_sync_slaves().", "fix_status": "fixed", "fix_kernel_uname": "DragonFly 6.5-DEVELOPMENT #1 (in-guest make nativekernel + DF-2657 fix.diff, 10 hunks; see fix_build.log.tgz)", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "VALIDATED: rebuilt kernel eliminates the wedge - h2idle parks (0 h2coll), umount -f RC=0 in 0s, third mount RC=0, 6/6 single-device churn control clean, diverged rw cluster tears down cleanly with the fresher master preserved (forward-only gate required). Residual DF-2620-family survivor UAF trips post-teardown (pre-existing, separately tracked).", "fix_evidence": "fix_run.log / fix_build.log.tgz / panic_serial.log (4 residual UAF traps attributed to DF-2620 family)" } |