DF-2630 / verdict.json
{ "finding_id": "DF-2630", "status": "reproduced", "reproduced": 1, "impact": "dos", "confidence": "certain", "verdict": "VERIFIED on stock INVARIANTS kernel: a single forged byte (PFS inode meta.pfs_nmasters 0x00->0x02, minimal realistic value, CHECK_NONE brefs + volhdr CRC recompute) on a single-device hammer2 image makes every mount-crossing operation wedge forever in hammer2_vfs_root (wchan h2root, 1 Hz PCATCH retry loop with no timeout/failure path at vfsops.c:1966-2011 because nquorum=2>nchains=1 can never be met); the first wedged lookups are signal-killable, but once umount -f takes the mount exclusively busy and itself wedges in teardown (h2twait), every further lookup piles up UNKILLABLE D-state in vfs_busy (observed 40+ minutes, kill -9 immune, pids 877/884), plain umount-by-path wedges in sys_unmount's nlookup, even a bare `mount` listing wedges via mountctl->nlookup, and shutdown is degraded ('init: some processes would not die', vn0 unmount fails BUSY, forced unmounts) - reboot is the only recovery. Causality proven by the one-byte-different control image (nmasters=1): mount/ls/umount all RC=0. Fix validated on rebuilt kernel #1: bounded (sysctl vfs.hammer2.root_timeout, default 60 s, 0=legacy) + sticky HAMMER2_PMPF_ROOTFAILED failure makes ls fail with EIO after the bound, retries fail instantly, zero h2root/vfs_busy waiters remain, umount -f cleanly REMOVES the bogus mount, further mounts work, control and stock images behave identically, and 280 s post-run soaks were clean. Residual documented: plain umount-by-path of the bogus mount fails EIO (path lookup fails once the root is marked failed) - umount -f removes it. vfs.usermount=0 on this guest, so mounts were root-position (same caveat as DF-2620).", "exploit_chain": "", "evidence": "run.log (steps 1-10 incl. MOUNT_RC=0 same-second, ls child wchan h2root, kill -9 survival, umount/umount -f wedges); dstate_census.txt (7 D-state processes wchan vfs_busy/h2twait 40+ min after kill -9); shutdown_serial.log (init: some processes would not die; unmount of /dev/vn0@testvol failed (BUSY); forced unmounts); fix_run.log (WEDGE_LS1 EIO after exactly 5 s, sticky instant fail, WEDGE_UMOUNTF_RC=0, dmesg 'root quorum cannot be reached (nmasters=2 nchains=1)'); control run CONTROL_*_RC=0", "kernel_refs": [ "sys/vfs/hammer2/hammer2_vfsops.c:527", "sys/vfs/hammer2/hammer2_vfsops.c:1966", "sys/vfs/hammer2/hammer2_vfsops.c:2007", "sys/vfs/hammer2/hammer2_cluster.c:348", "sys/kern/vfs_nlookup.c:1053", "sys/kern/vfs_cache.c:4504", "sys/kern/vfs_syscalls.c:635" ], "poc_changes": "Clean standalone forger derived from DF-2620's forge (same CHECK_NONE/CRC technique) with a minimal realistic nmasters=2 variant plus a one-byte-different nmasters=1 control; trigger script hardened after the first run (bare `mount` replaced by `mount -p` because prmount->mountctl->nlookup itself wedges on the mountpoint - accidentally proven, mount(8) pid 940 wchan h2root).", "attempts": 3, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 (baseline) / #1: Sat Aug 29 05:10:55 UTC 2026 (fix), x86_64", "runtime_sec": 9600, "guest_dirty": 0, "build_cmd": "python3 forge_df2630.py base.img wedge wedge.img; python3 forge_df2630.py base.img control control.img; (fix kernel) cd /usr/src && make nativekernel && make installkernel", "run_cmd": "sh trigger_df2630.sh # stock wedge; sh control_df2630.sh # causality; sh fix_validate.sh # fix kernel", "code_hash": "8c4c328401248c1cefb70d27914e2f3b787e631fda13effd49c237595e0817d6", "notes": "Anomaly recorded honestly: one early `stat /mnt/h2` returned the covered directory's attributes without wedging (hashed root-fs inum 2501377919); all operations that must cross into the mount wedge permanently. Full untrimmed nativekernel build logs were lost to QEMU disk reverts between fix iterations (BUILD_RC=0 recorded per build). Guest left clean (vm.sh reset with-src).", "recommended_fix": "Bound hammer2_vfs_root's quorum wait with sysctl vfs.hammer2.root_timeout (default 60 s, 0=legacy) and fail VFS_ROOT with EIO, remembering the failure in HAMMER2_PMPF_ROOTFAILED so subsequent lookups fail immediately; see fix.diff.", "fix_status": "fixed", "fix_kernel_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #1: Sat Aug 29 05:10:55 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "On the rebuilt kernel the previously-observed bad behavior is GONE: wedge-image ls fails EIO after the (5 s demo) bound instead of wedging forever, the sticky flag makes retries instant, no h2root/vfs_busy/ncplk D-state pile forms, `umount -f` returns 0 and removes the bogus mount, and stock/control images are unaffected (mount/ls/umount all RC=0). 280 s post-run soaks clean, zero leftover threads, no panics. The 2630 hunks are byte-identical across all validated fix builds.", "fix_evidence": "fix_run.log (SANITY/CONTROL RC=0, WEDGE_LS1_RC=1 after exactly 5 s with 'Input/output error', WEDGE_LS2_RC=1 same-second sticky, CENSUS_DONE zero waiters, dmesg quorum message, WEDGE_UMOUNTF_RC=0, HEALTH_MOUNT_RC=0); fix.diff" } |