DF-2657 — hammer2 sync-thread xop quorum counts the excluded node: every

h2nod collect can never reach quorum (root cause of the DF-2631 wedge family)

sys/vfs/hammer2/hammer2_synchro.c (hammer2_sync_slaves, lines 411/454/672 — the hammer2_xop_start_except(..., idx) call sites) interacting with sys/vfs/hammer2/hammer2_cluster.c:348/462-465/522-527 (quorum math) and sys/vfs/hammer2/hammer2_admin.c:888-998 (hammer2_xop_collect).

Medium / local DoS (same impact class as DF-2631: wedged sync threads, stalled/failed umount -f, mntlk held, frozen thread cohorts, shutdown degradation). Bucket: hammer2. This is the precise, fixable root cause that DF-2631 (fix: inconclusive) asked for, plus the missing direction gate that any unwedging fix needs to avoid destructive regression.

Root cause (path:line)

• hammer2_sync_slaves() issues all three of its xops with hammer2_xop_start_except(&xop->head, desc, idx) — its own cluster index is never queued and can never feed (synchro.c:411, 454, 672).
• hammer2_cluster_check() computes the quorum over the whole cluster: nquorum = pmp->pfs_nmasters / 2 + 1 (cluster.c:348), and in pass 2 the excluded element (chain==NULL, no CITEM_NULL) is counted as an "in-progress master": ++umasters (cluster.c:462-465).
• With nmasters + umasters >= nquorum (cluster.c:523) the check returns EINPROGRESS forever — the excluded node's vote is awaited eternally.
• 2-node MASTER+MASTER (nmasters=2, nquorum=2, participants=1): 1 collected + 1 phantom = 2 >= 2 → EINPROGRESS forever.
• MASTER+SLAVE where idx is the master (nquorum=1, participants have 0 masters): 0 + 1 >= 1 → EINPROGRESS forever.
• 1-element non-MASTER pmppfs (the boot "LOCAL" PFS on every mounted device): 0 participants → 0 + 1 >= 1 → EINPROGRESS forever.
• hammer2_xop_collect() then loops forever (tsleep "h2coll" hz*60, admin.c:990-998) with no hammer2_thr_break()/STOP/FREEZE check inside the loop, so the sync thread can never be stopped or frozen cleanly → hammer2_thr_freeze()/hammer2_thr_delete() block in h2twait (admin.c:148/259) → the DF-2631 umount -f wedge + frozen cohorts.

Net effect: hammer2 slave/master resynchronization has never functioned on any cluster whose participating masters are fewer than pfs_nmasters/2+1 — every such sync thread parks in h2coll before its first comparison.

Fix (fix.diff, validated in-guest)

Participant-aware quorum + forward-only sync gate:

1. hammer2_xop_start_except() marks the xop HAMMER2_XOP_PARTQUORUM (notidx >= 0) and rebuilds xop->chk_mask from scratch (= participants).
2. hammer2_xop_collect() passes the participant mask to hammer2_cluster_check().
3. hammer2_cluster_check() computes nquorum from participating MASTER/SUPROOT elements only and skips non-participants in pass 2 (they can neither vote nor be "in progress"). Normal all-node xops keep bit-for-bit legacy behavior (pmask = ~0).
4. hammer2_sync_slaves() gate becomes forward-only: want_update = (chain->bref.modify_tid < sync_tid) (synchro.c:421) — without this, unwedged 2-master clusters would ping-pong destructively (each thread dragging its column toward the other's older tid).

Reproduce (guest, root)

sh trigger_df2657.sh     # stock: 4x h2nod in h2coll pre-umount;
                        # umount -f D4 h2twait 65 s -> rc=1, stranded
                        # threads; third mount D3 mntlk.

Fix validation (kernel #1, make nativekernel && make installkernel)

sh fix_validate.sh      # Phase A: h2idle parks (0x h2coll), umount -f
                        # RC=0 in 0 s, third mount instant, single-device
                        # sanity OK; control: 6x solo cycles clean.
                        # Phase B: diverged rw cluster, no regression,
                        # clean teardown.  Phase C: post-cluster-teardown
                        # mount churn trips the pre-existing DF-2620-
                        # family UAF (same +0x2d9 trap as DF-2631's
                        # builds) — see VERDICT.md.

Files

trigger_df2657.sh    stock baseline (reconstructed log: run.log)
fix_validate.sh      validation battery (reconstructed log: fix_run.log)
fix.diff             participant-quorum + forward-only gate (applies -p1)
fix_build.log.tgz    FULL untrimmed nativekernel+installkernel log (5.4 MB)
panic_serial.log     the 4 DF-2620-family traps on the fix kernel
fix_run.truncated.log in-guest log frozen by panic #1 (23 lines)
code_hashes.txt      sha256 of diff/scripts

DF-2657 — participant-blind quorum parks every hammer2 sync thread (h2coll)

sys/vfs/hammer2/hammer2_synchro.c (sync xop issue sites 411/454/672) x sys/vfs/hammer2/hammer2_cluster.c (cluster_check quorum math) x sys/vfs/hammer2/hammer2_admin.c (xop_collect EINPROGRESS loop)

CLAIM VERIFIED (reproduced) on the stock INVARIANTS kernel, unmodified dd-clone images, and the wedge ELIMINATED on a rebuilt kernel carrying fix.diff (kernel #1). This finding supplies the precise fixable root cause requested by DF-2631 (whose own timeout-bound fix was inconclusive) and is filed as a distinct defect at the fix level: the quorum computation itself, not a timeout around it.

REPRODUCED (stock kernel #0) 1. m1/m2 = dd clones of one newfs_hammer2 -L testvol image (identical MD5), MASTER+MASTER, same pfs_clid. 2. mount -o ro vn0@testvol (RC=0); mount vn1@testvol -> EBUSY but the chain merges into the pmp (nmasters 1->2). 3. Census BEFORE any umount: 4 sync threads parked in h2coll (2x h2nod-testvol, 2x h2nod-LOCAL) — the permanent park. This is the defect itself: the collect can never reach quorum because the excluded own index is counted as an eternally-in-progress master (cluster.c:462-465) against nquorum = pfs_nmasters/2+1 (cluster.c:348). 4. umount -f: umount(8) D4 h2twait for ~65 s (one hz*60 tsleep cycle spanning freeze waits), then rc=1 with EMPTY error message; the 4 h2nod threads remain in h2coll FOREVER (census minutes later), frozen h2xop cohorts remain, the pmp is dead weight. 5. Third same-clid mount: D3 mntlk (hammer2_mntlk held by the stuck umount syscall), times out. => local DoS: any user who can induce an admin to mount a same-clid clone pair (or any genuine >=2-master / master+slave cluster, where the same math parks the sync threads from the moment of the merge) gets a wedgeable mount that cannot be unmounted cleanly.

ROOT CAUSE (path:line) * synchro.c:411/454/672 — hammer2_sync_slaves() issues ipcluster/ scanall xops via hammer2_xop_start_except(..., idx): own index never queued, never feeds (admin.c:556-571 skips i == notidx). * cluster.c:348 — nquorum = pmp->pfs_nmasters/2+1 over ALL masters. * cluster.c:462-465 — the excluded element (NULL chain, no CITEM_NULL) counts as ++umasters ("in progress") forever. * cluster.c:522-527 — nmasters + umasters >= nquorum => EINPROGRESS forever for: 2xMASTER (2>1), MASTER(idx)+SLAVE (1>0), 1-element non-MASTER pmp such as the boot "LOCAL" PFS (1>0). * admin.c:990-998 — collect loops on EINPROGRESS (tsleep "h2coll" hz*60) with NO thr_break/STOP/FREEZE check => the thread can never be stopped or frozen => admin.c:148/259 h2twait wedge (DF-2631).

FIX (fix.diff) — participant-aware quorum + forward-only gate a. hammer2.h: HAMMER2_XOP_PARTQUORUM (0x20) + cluster_check takes pmask. b. hammer2_admin.c hammer2_xop_start_except(): set PARTQUORUM when notidx >= 0; rebuild xop->chk_mask under xop_spin (exactly the queued participants; chk_mask is otherwise never reset — my clear also fixes stale bits surviving objcache recycling). c. hammer2_admin.c hammer2_xop_collect(): pmask = chk_mask for PARTQUORUM xops, else ~0 (legacy, bit-for-bit). d. hammer2_cluster.c hammer2_cluster_check(): nquorum from participating MASTER/SUPROOT elements (npmasters/2+1; pmp==NULL degenerate preserved); pass 2 skips non-participants. e. hammer2_synchro.c:421: want_update = (mtid < sync_tid) — forward- only. REQUIRED companion: with quorum now reachable among peers, a diverged peer with an OLDER root tid must not drag a newer column backward (otherwise a diverged 2-master cluster ping-pongs destroy/recreate between passes).

FIX VALIDATION (kernel #1, Sun Aug 30 04:30:11 UTC 2026) Build: cd /usr/src && make nativekernel && make installkernel -> BUILD_INSTALL_OK; full 5.4 MB log kept (fix_build.log.tgz). Apply: patch -p1 < fix.diff — all 10 hunks clean. Phase A (identical clones, the DF-2631 scenario): h2nod census post-merge: 4x h2idle, A_H2COLL_COUNT=0 (stock: 4x h2coll forever); ls OK (VFS quorum path unchanged); umount -f: RC=0 in 0 s (stock: rc=1 after 65 s + stranded threads); third mount: RC=0 instant (stock: mntlk D3); single-device sanity mount/umount: RC=0/RC=0. Control: 6x solo mount/ls/umount churn on one boot: all RC=0, guest up, no panics => the fix does not disturb single-device pmp lifecycle. Phase B (diverged rw cluster, m1 has X / m2 not): merge OK; 4x h2idle; no "hammer2_sync_slaves: error" or "no quorum possible" spam; plain umount RC=0 in 0 s (stock: impossible); b1 (fresher) still contains X after the session => NO REGRESSION (forward-only gate holds). Pull-in of X into the stale clone's media was NOT observed — see Follow-ups. Phase C (stability): any session combining a cluster merge+teardown with 1-3 subsequent same-clid mount operations panicked at hammer2_primary_xops_thread+0x2d9 (lock xaddl %edx, 0x81558(%rsi)) — 4 occurrences (trap 12 and trap 1). This is the PRE-EXISTING DF-2620-family multi-chain teardown UAF: identical trap site+instruction to the delayed panics on DF-2631's independent (timeout-based) fix builds, and the no-cluster control is clean. Stock never reaches it because the umount wedges first. NOT introduced by this fix; it is the next defect in the teardown chain this fix uncovers.

FIX STATUS: fixed (for the filed defect) — the h2coll park, the umount -f stall/failure, the mntlk hold and the frozen cohorts are all eliminated on the patched kernel; single-device and VFS-quorum behavior unchanged. Full-system stability after cluster teardown remains blocked by the separate, pre-existing DF-2620-family survivor-worker UAF (documented, out of scope here).

FOLLOW-UPS (open, honestly noted) 1. DF-2620 family: crash-dump analysis of the +0x2d9 survivor cohort after cluster teardown (xop_groups lifetime vs pfsfree_scan). 2. Stale-column pull-in: with participant quorum the stale column's sync pass completes without error yet X did not appear on m2's media (b1/b2 MD5s differ; b1 keeps X). Suspects: flush semantics of sync-thread-created chains on the second column, or the non-modifying trans flags the sync thread runs under (hammer2_trans_init(pmp, 0), synchro.c:178). Needs vfs.hammer2.debug=0x8000 tracing on a UAF-free kernel (every diagnostic attempt on this guest was killed by the DF-2620 trap).

Recovery on stock: reboot only (umount never completes cleanly).