DF-2631 — umount -f of a same-clid multi-device hammer2 cluster hangs (h2twait)

Medium / local DoS (plus an exposed pre-existing teardown UAF — see VERDICT.md "FIX STATUS"). Two dd clones of one newfs_hammer2 image (MASTER+MASTER, same pfs_clid), mount one, attempt the second (EBUSY, chain merges), then umount -f → the umount thread wedges forever in hammer2_thr_wait (wchan h2twait), hammer2_mntlk stays held (later hammer2 mounts block in mntlk), and shutdown hangs.

Reproduce (guest, root; unmodified images)

sh trigger_df2631.sh    # census before/after; leaves guest wedged

Expected (stock): umount -f pid D-state wchan h2twait; both h2nod-testvol sync threads wchan h2coll; ~108 h2xop-testvol survivors (frozen h2twait / mixed h2idle generations); third mount mount_hammer2 D-state wchan mntlk; serial: "init: some processes would not die", vn0 unmount fails BUSY. Recovery: reboot only.

Fix validation (kernel rebuilt with fix.diff)

sh fix_validate.sh      # cluster umount -f must COMPLETE (bounded)

Expected (fixed, defaults): UMOUNTF_RC=0 after ~61 s (ELAPSED=61s; with vfs.hammer2.xop_collect_timeout=10 it is ~11 s); third mount/ls/umount RC=0; single-device sanity RC=0.

Fix status: inconclusive — the wedge is eliminated, but the unwound teardown reproducibly triggers a delayed (1-5 min) page fault in hammer2_primary_xops_thread (pre-existing DF-2620-family UAF that stock never reaches because it hangs first). See VERDICT.md.

Files

trigger_df2631.sh    stock-kernel wedge run + censuses
fix_validate.sh      fix-kernel validation battery
run.log              stock trigger output (reconstructed verbatim from
                    the session; decisive lines only — see note inside)
fix_run.log          fix validation output (kernel #1, 61 s run +
                    earlier 11 s run), reconstructed from session
shutdown_serial.log  degraded-shutdown serial console (raw)
fix_iteration{1,2,3}_panic_serial.log  serial of the delayed panics on
                    intermediate fix builds (root-cause artifacts)
fix.diff             freeze reorder + teardown guard + bounded collect (per-finding, applies to pristine)
fix.combined.diff    BOTH findings' fixes in one diff = the exact tree built
                     and validated in the guest (use this to reproduce
                     the fix kernel; per-finding diffs touch adjacent
                     lines in hammer2.h/vfsops.c, so apply the combined
                     one or use `patch -l` when stacking them)
env.txt              kernels, sysctls, image md5s
manifest.json        machine-readable catalog
verdict.json         machine verdict

DF-2631 — umount -f of a same-clid multi-device hammer2 cluster hangs in hammer2_thr_wait ("h2twait") on unmodified images ======================================================================= sys/vfs/hammer2/hammer2_admin.c (hammer2_thr_wait / xop machinery) + sys/vfs/hammer2/hammer2_vfsops.c (pfsfree_scan / vfs_unmount teardown)

CLAIM VERIFIED (reproduced) on the stock INVARIANTS kernel with UNMODIFIED images (dd clones of one newfs_hammer2 image, MASTER+MASTER, same pfs_clid). Fix validation: the wedge is ELIMINATED on the rebuilt kernel (umount -f completes, bounded), but a residual pre-existing UAF in the multi-chain teardown becomes reachable — see "Fix status" below.

Reproduction (stock kernel #0): 1. m1.img/m2.img = dd clones of one newfs_hammer2 -L testvol image (identical md5 956fc18a70f6689a8669e21d372b4eb9). 2. mount -o ro /dev/vn0@testvol /mnt/h2 -> RC=0 (36 h2xop workers, 0 sync threads: single master). 3. mount -o ro /dev/vn1@testvol /mnt/h2b -> EBUSY ("PFS already mounted!") BUT the second chain merged into the same pmp at clindex 1: 108 h2xop-testvol workers + 2 h2nod-testvol sync threads now exist; nmasters = 2 (count bump, vfsops.c:536-542). 4. ls /mnt/h2 -> works (quorum 2/2). 5. umount -f /mnt/h2 -> D-state, wchan "h2twait", FOREVER (observed at +3s, +8s, +63s; no progress across 60-second tsleep cycles).

Thread census at the wedge (stock): - umount -f pid 932: D3 h2twait (hammer2_thr_freeze waiting for a sync thread to reach FROZEN) - 2 x h2nod-testvol: h2coll (sync threads parked in hammer2_xop_collect, EINPROGRESS loop, no failure path) - 108 x h2xop-testvol: h2twait (frozen workers inside thr_wait_any(UNFREEZE|STOP, 0)) — plus a second mixed generation (some h2idle): xop_groups was lazily RE-CREATED mid-teardown. - third mount (vn2): mount_hammer2 D3 mntlk (hammer2_mntlk held by the wedged umount syscall). - shutdown -r: "init: some processes would not die", vn0 unmount fails BUSY, forced unmounts (shutdown_serial.log).

Root cause (path:line): 1. sys/vfs/hammer2/hammer2_synchro.c:411/454 — each sync thread's hammer2_sync_slaves() issues its xops with hammer2_xop_start_except(..., idx): its OWN cluster index is never queued and never feeds. For nmasters=2 the collect's quorum (cluster.c:348 nquorum = nmasters/2+1 = 2) is therefore unreachable forever (own column counted as 'umasters', admin.c:990-998 EINPROGRESS -> tsleep "h2coll" hz*60 -> retry). The sync threads of EVERY >=2-master cluster park in h2coll (observed pre-umount in the T2 census). 2. sys/vfs/hammer2/hammer2_vfsops.c:1664 hammer2_xop_helper_cleanup() deletes all xop workers while those sync threads are still alive mid-collect; surviving issuers re-create the whole xop_groups array lazily (admin.c:491-492), leaving worker threads inside arrays/scans that pfsfree_scan() is ripping apart. 3. sys/vfs/hammer2/hammer2_vfsops.c:772-793 pfsfree_scan()'s freeze phase freezes xop columns interleaved with the sync threads; a sync thread parked in a collect that can never be fed never observes FREEZE (checked only at its loop top, synchro.c:115/127), so hammer2_thr_freeze() (admin.c:288-294, hammer2_thr_wait "h2twait") blocks forever. umount holds hammer2_mntlk (vfsops.c:1640): all later hammer2 mounts block (wchan mntlk) and shutdown hangs the same way.

Fix (fix.diff) — three cooperating changes: a. hammer2_xop_collect(): elapsed-time bound (vfs.hammer2.xop_collect_timeout, default 60 s, 0 = legacy) on the EINPROGRESS wait, EXEMPTing super-root (SPMP) pmps whose sync threads legitimately run long scans. Event-counted bounds are defeated by continuous feeds from the healthy column (measured!), so the bound is elapsed-based. b. hammer2_pfsfree_scan(): freeze ORDER fixed — all sync threads are brought to FROZEN before any xop helper column is frozen (a frozen helper column starves in-progress collects issued by sync threads). c. hammer2_xop_start_except(): HAMMER2_PMPF_TEARDOWN guard — once hammer2_vfs_unmount() has started, no lazy re-creation of xop helpers; the starter instead feeds an immediate EOF+EIO for every node so the frontend collect fails cleanly in one pass. This eliminates the "cohort re-created mid-teardown" UAF source (observed as 36-thread survivor cohorts and Fatal trap 12 in hammer2_primary_xops_thread+0x2d9 on intermediate fix builds).

Fix validation (kernel #1): - Cluster umount -f COMPLETES: RC=0 in 61 s at the default timeout (11 s with the knob at 10) — versus infinite hang on stock. - Third mount of a fresh clone works (mntlk released), single-device mount/umount unaffected, no wedged threads immediately after.

FIX STATUS: INCONCLUSIVE (wedge fixed, adjacent latent UAF exposed). On every kernel carrying the working (elapsed) collect bound, a delayed destabilization followed the cluster umount -f 1-5 minutes later: Fatal trap 12 page fault at hammer2_primary_xops_thread+0x2d9 (lock xaddl %edx,0x81558(%rsi) — a worker touching freed xop/thread state) or a spontaneous reset (no dump). This is a PRE-EXISTING memory-corruption defect of the multi-chain teardown (same family as DF-2620: xop worker lifetime vs hammer2_xop_helper_cleanup/ pfsfree_scan teardown) that stock never reaches because the umount wedges first. Fix iterations v1 (pfsfree kfree of xop_groups), v2/v3/v4 (teardown guard + feed-EOF), v5/v6 (elapsed bound, SPMP exemption), v7 (teardown-gated bound — did not unwedge) are documented in fix_iteration{1,2,3}_panic_serial.log. Follow-up required: crash-dump analysis of the surviving worker cohort and a lifetime audit of hammer2_xop_start*() vs teardown; until then the DoS-wedge is traded for a visible (panic-once) corruption instead of a silent hang.

Recovery on stock: reboot only (umount -f never completes; mntlk held).