DF-2631 — umount -f of a same-clid multi-device hammer2 cluster hangs
in hammer2_thr_wait ("h2twait") on unmodified images
=======================================================================
sys/vfs/hammer2/hammer2_admin.c (hammer2_thr_wait / xop machinery) +
sys/vfs/hammer2/hammer2_vfsops.c (pfsfree_scan / vfs_unmount teardown)

CLAIM VERIFIED (reproduced) on the stock INVARIANTS kernel with
UNMODIFIED images (dd clones of one newfs_hammer2 image, MASTER+MASTER,
same pfs_clid).  Fix validation: the wedge is ELIMINATED on the rebuilt
kernel (umount -f completes, bounded), but a residual pre-existing UAF
in the multi-chain teardown becomes reachable — see "Fix status" below.

Reproduction (stock kernel #0):
  1. m1.img/m2.img = dd clones of one newfs_hammer2 -L testvol image
     (identical md5 956fc18a70f6689a8669e21d372b4eb9).
  2. mount -o ro /dev/vn0@testvol /mnt/h2        -> RC=0 (36 h2xop
     workers, 0 sync threads: single master).
  3. mount -o ro /dev/vn1@testvol /mnt/h2b       -> EBUSY ("PFS already
     mounted!") BUT the second chain merged into the same pmp at
     clindex 1: 108 h2xop-testvol workers + 2 h2nod-testvol sync
     threads now exist; nmasters = 2 (count bump, vfsops.c:536-542).
  4. ls /mnt/h2 -> works (quorum 2/2).
  5. umount -f /mnt/h2 -> D-state, wchan "h2twait", FOREVER (observed
     at +3s, +8s, +63s; no progress across 60-second tsleep cycles).

Thread census at the wedge (stock):
  - umount -f pid 932:            D3  h2twait   (hammer2_thr_freeze
      waiting for a sync thread to reach FROZEN)
  - 2 x h2nod-testvol:            h2coll       (sync threads parked in
      hammer2_xop_collect, EINPROGRESS loop, no failure path)
  - 108 x h2xop-testvol:          h2twait      (frozen workers inside
      thr_wait_any(UNFREEZE|STOP, 0)) — plus a second mixed generation
      (some h2idle): xop_groups was lazily RE-CREATED mid-teardown.
  - third mount (vn2):           mount_hammer2 D3 mntlk (hammer2_mntlk
      held by the wedged umount syscall).
  - shutdown -r: "init: some processes would not die", vn0 unmount
    fails BUSY, forced unmounts (shutdown_serial.log).

Root cause (path:line):
  1. sys/vfs/hammer2/hammer2_synchro.c:411/454 — each sync thread's
     hammer2_sync_slaves() issues its xops with
     hammer2_xop_start_except(..., idx): its OWN cluster index is never
     queued and never feeds.  For nmasters=2 the collect's quorum
     (cluster.c:348 nquorum = nmasters/2+1 = 2) is therefore
     unreachable forever (own column counted as 'umasters',
     admin.c:990-998 EINPROGRESS -> tsleep "h2coll" hz*60 -> retry).
     The sync threads of EVERY >=2-master cluster park in h2coll
     (observed pre-umount in the T2 census).
  2. sys/vfs/hammer2/hammer2_vfsops.c:1664 hammer2_xop_helper_cleanup()
     deletes all xop workers while those sync threads are still alive
     mid-collect; surviving issuers re-create the whole xop_groups
     array lazily (admin.c:491-492), leaving worker threads inside
     arrays/scans that pfsfree_scan() is ripping apart.
  3. sys/vfs/hammer2/hammer2_vfsops.c:772-793 pfsfree_scan()'s freeze
     phase freezes xop columns interleaved with the sync threads; a
     sync thread parked in a collect that can never be fed never
     observes FREEZE (checked only at its loop top,
     synchro.c:115/127), so hammer2_thr_freeze() (admin.c:288-294,
     hammer2_thr_wait "h2twait") blocks forever.  umount holds
     hammer2_mntlk (vfsops.c:1640): all later hammer2 mounts block
     (wchan mntlk) and shutdown hangs the same way.

Fix (fix.diff) — three cooperating changes:
  a. hammer2_xop_collect(): elapsed-time bound
     (vfs.hammer2.xop_collect_timeout, default 60 s, 0 = legacy) on the
     EINPROGRESS wait, EXEMPTing super-root (SPMP) pmps whose sync
     threads legitimately run long scans.  Event-counted bounds are
     defeated by continuous feeds from the healthy column (measured!),
     so the bound is elapsed-based.
  b. hammer2_pfsfree_scan(): freeze ORDER fixed — all sync threads are
     brought to FROZEN before any xop helper column is frozen (a frozen
     helper column starves in-progress collects issued by sync
     threads).
  c. hammer2_xop_start_except(): HAMMER2_PMPF_TEARDOWN guard — once
     hammer2_vfs_unmount() has started, no lazy re-creation of xop
     helpers; the starter instead feeds an immediate EOF+EIO for every
     node so the frontend collect fails cleanly in one pass.  This
     eliminates the "cohort re-created mid-teardown" UAF source
     (observed as 36-thread survivor cohorts and Fatal trap 12 in
     hammer2_primary_xops_thread+0x2d9 on intermediate fix builds).

Fix validation (kernel #1):
  - Cluster umount -f COMPLETES: RC=0 in 61 s at the default timeout
    (11 s with the knob at 10) — versus infinite hang on stock.
  - Third mount of a fresh clone works (mntlk released), single-device
    mount/umount unaffected, no wedged threads immediately after.

FIX STATUS: INCONCLUSIVE (wedge fixed, adjacent latent UAF exposed).
  On every kernel carrying the working (elapsed) collect bound, a
  delayed destabilization followed the cluster umount -f 1-5 minutes
  later: Fatal trap 12 page fault at hammer2_primary_xops_thread+0x2d9
  (lock xaddl %edx,0x81558(%rsi) — a worker touching freed xop/thread
  state) or a spontaneous reset (no dump).  This is a PRE-EXISTING
  memory-corruption defect of the multi-chain teardown (same family as
  DF-2620: xop worker lifetime vs hammer2_xop_helper_cleanup/
  pfsfree_scan teardown) that stock never reaches because the umount
  wedges first.  Fix iterations v1 (pfsfree kfree of xop_groups),
  v2/v3/v4 (teardown guard + feed-EOF), v5/v6 (elapsed bound, SPMP
  exemption), v7 (teardown-gated bound — did not unwedge) are
  documented in fix_iteration{1,2,3}_panic_serial.log.  Follow-up
  required: crash-dump analysis of the surviving worker cohort and a
  lifetime audit of hammer2_xop_start*() vs teardown; until then the
  DoS-wedge is traded for a visible (panic-once) corruption instead of
  a silent hang.

Recovery on stock: reboot only (umount -f never completes; mntlk held).