DF-2630 — hammer2 VFS_ROOT quorum wedge (pfs_nmasters > nchains)

Medium / local DoS. A single forged byte (PFS inode meta.pfs_nmasters 0x00 -> 0x02) on a single-device hammer2 image makes every operation that crosses into the mount wedge forever in hammer2_vfs_root (wchan h2root), and — after umount -f wedges in teardown — every further lookup on the mountpoint piles up unkillable D-state (vfs_busy), the mount can never be unmounted, and shutdown is degraded. Fix (bounded + sticky-fail VFS_ROOT) validated on a rebuilt kernel.

Reproduce

1. Forge the images (host, needs python3)

# base image created once in the guest:
#   truncate -s 64M base.img && newfs_hammer2 -L testvol base.img
python3 forge_df2630.py base.img wedge   wedge.img   # nmasters=2 (wedge)
python3 forge_df2630.py base.img control control.img # nmasters=1 (control)
cmp -l wedge.img control.img     # exactly ONE byte differs (0x1400886)

2. Run in the guest (root)

sh trigger_df2630.sh        # wedge + full census (leaves guest wedged)
# separate boot:
sh control_df2630.sh        # causality control: mount/ls/umount all RC=0

Expected (stock kernel): ls /mnt/h2 child in S-state wchan h2root; later lookups + umount in D-state wchan vfs_busy (survive kill -9); umount -f in D-state wchan h2twait; serial console at shutdown: "init: some processes would not die", "unmount ... failed (BUSY)". Recovery: reboot only (umount -f does not complete on stock).

3. Fix validation (kernel rebuilt with fix.diff)

sh fix_validate.sh          # sanity + control + wedge + health + umount -f

Expected (fixed kernel, root_timeout=5 demo): ls #1 fails EIO after ~5 s; ls #2 fails instantly (sticky); zero h2root/vfs_busy/ ncplk waiters; umount -f /mnt/h2 returns 0 and removes the mount; fresh hammer2 mounts elsewhere keep working.

Files

forge_df2630.py      image forger (wedge | control) — host python3
trigger_df2630.sh    stock-kernel wedge run + census
control_df2630.sh    control-image causality run
fix_validate.sh      fix-kernel validation battery
run.log              stock-kernel trigger output (full, untrimmed)
dstate_census.txt    unkillable D-state pile-up (reconstructed from the
                     live session; the guest files were lost to resets)
shutdown_serial.log  serial console of the degraded shutdown (raw)
fix_run.log          fixed-kernel validation output (kernel #1, v4)
fix.diff             the validated fix (hammer2.h + hammer2_vfsops.c)
env.txt              kernels, sysctls, image md5s
manifest.json        machine-readable catalog
verdict.json         machine verdict

NOTE on applying fixes: DF-2630's and DF-2631's per-finding fix.diff files touch adjacent lines in hammer2.h and hammer2_vfsops.c; each applies cleanly to the pristine tree on its own, but stacking them sequentially needs patch -l (or apply fix.combined.diff, which is the exact tree built and validated in the guest).

DF-2630 — hammer2 VFS_ROOT quorum wedge: pfs_nmasters > nchains pins the mount and produces an unkillable D-state pile-up ========================================================================= sys/vfs/hammer2/hammer2_vfsops.c:1966-2011 (stock tree)

CLAIM VERIFIED (reproduced) on the stock INVARIANTS kernel, and the fix VALIDATED on a rebuilt kernel (kernel #1).

What was verified (stock kernel, wedge.img = newfs image with ONE byte patched: testvol PFS inode meta.pfs_nmasters 0x00 -> 0x02; CHECK_NONE brefs + volhdr CRC recompute; control.img differs from wedge.img by exactly that one byte: 0x01):

1. mount(2) itself RETURNS (vfsops.c:1949 hammer2_vfs_root is not called by the mount syscall; VFS_ROOT fires on the first lookup crossing). MOUNT_RC=0, same-second timestamps.
2. First ls /mnt/h2 wedges in hammer2_vFS_root: child process in S-state, wchan "h2root" (the PCATCH tsleep at vfsops.c:2007, 1 Hz retry loop with NO timeout and NO failure path while pmp->inode_tid == 0).
3. Later lookups degenerate into UNKILLABLE D-state: after umount -f takes the mount exclusively busy and then itself wedges, every new crossing lookup blocks in vfs_busy (nlookup crossing loop, sys/kern/vfs_nlookup.c:1053-1062; cache_resolve_mp, sys/kern/vfs_cache.c:4504-4507). Observed: pids 877/884 (orphaned ls) in D-state wchan vfs_busy for 40+ MINUTES, immune to kill -9.
4. umount /mnt/h2 (plain) wedges: sys_unmount() nlookups the path (sys/kern/vfs_syscalls.c:635-638) -> crossing -> VFS_ROOT/wedge; observed D-state wchan vfs_busy.
5. umount -f /mnt/h2 wedges in hammer2 teardown (D-state, wchan h2twait) - the wedged VFS_ROOT loop keeps re-issuing ipcluster xops every second, and the teardown waits never complete.
6. Even a bare mount listing hangs (prmount -> mountctl(f_mntonname) -> nlookup -> VFS_ROOT): accidentally proven in the first run (mount(8) pid 940, wchan h2root).
7. Shutdown is degraded: serial console shows "init: some processes would not die", "unmount of filesystem mounted from /dev/vn0@testvol failed (BUSY)", forced unmounts; the box only goes down via the boot()-time forced-unmount path. shutdown_serial.log.
8. Causality proof: control.img (pfs_nmasters=1, one byte different) mounts, lists and unmounts perfectly (CONTROL_MOUNT_RC=0, CONTROL_LS_RC=0, CONTROL_UMOUNT_RC=0).
9. Unprivileged reachability: vfs.usermount=0 on this guest; with usermount=1 + an owned vn/mem device the same image would be mountable by an unprivileged user (same position as DF-2620: root-position-only on this build, recorded).

Anomaly noted honestly: one early stat /mnt/h2 returned the covered directory's attributes without wedging (inum 2501377919 = hashed root-fs inum); operations that must CROSS into the mount (opendir/ readdir, umount by path, mountctl) wedge permanently.

Root cause chain (path:line): - sys/vfs/hammer2/hammer2_vfsops.c:527-528 ingests on-disk ripdata->meta.pfs_nmasters verbatim (uint8_t, no clamp, no reconciliation against the number of attached chains). - sys/vfs/hammer2/hammer2_cluster.c:348 nquorum = pfs_nmasters/2+1 = 2

nchains = 1: hammer2_cluster_check() can never reach quorum (ESRCH/EINPROGRESS forever). - sys/vfs/hammer2/hammer2_vfsops.c:1966-2011 loops forever (1 Hz tsleep "h2root"), holding the iroot lock across iterations and pinning whatever locks the caller holds (vfs_busy ref; ncp lock in the cache_resolve_mp path).

Fix (fix.diff, validated): - New sysctl vfs.hammer2.root_timeout (default 60 s, 0 = legacy infinite): hammer2_vfs_root bounds its total quorum wait and fails with EIO; a sticky HAMMER2_PMPF_ROOTFAILED flag makes subsequent VFS_ROOT calls fail immediately instead of re-waiting on every lookup. - Validated on the rebuilt kernel: wedge.img ls fails with EIO after the (5 s demo) timeout, second ls fails instantly, ZERO h2root/ vfs_busy/ncplk waiters, dmesg shows "root quorum cannot be reached (nmasters=2 nchains=1)", umount -f cleanly REMOVES the wedged mount (RC=0), further hammer2 mounts work, and the control + stock images behave identically to stock. 280-second post-run soaks on the fixed kernel were clean (no panics, no leftover threads). - Residual (documented): the bogus mount itself cannot be removed by a plain umount by path (the path lookup fails EIO once the root is marked failed), but umount -f removes it and shutdown unmounts it; the system otherwise stays fully healthy.