DF-2832 / verdict.json
{ "finding_id": "DF-2832", "status": "reproduced", "reproduced": 1, "impact": "panic", "confidence": "certain", "verdict": "The syncer-trigger API (trigger_syncer/trigger_syncer_start/trigger_syncer_stop/speedup_syncer, sys/kern/vfs_sync.c:561-614) loads mp->mnt_syncer_ctx lock-free and atomically RMWs the ctx with no lifetime interlock against vn_syncer_thr_stop() (vfs_sync.c:353-357) which NULLs the pointer and hashdestroy+kfree's the ctx. Demonstrated on the guest: six unprivileged (nobody) trigger callers -- stalled in hammer2_pfs_memory_wait via open(O_CREAT)->ncp_writechk->VFS_MODIFYING -- were held at the exact stock load->RMW instruction boundary while the REAL vn_syncer_thr_stop() freed the ctx under them; they then observed the teardown ('DF2832: HIT stale ctx=0x...'), i.e. on unmodified stock code their atomic_fetchadd_int(&ctx->syncer_trigger,2) would have executed on freed heap. The same run panicked at lwkt_gettoken+0x64 via the adjacent NULL-ctx facet (vn_syncer_add on the torn-down-but-live mount) -- that facet is not reachable through the stock umount ordering (VFS_UNMOUNT vflushes first), but the freed-ctx RMW facet is, via umount -f racing any in-flight hammer2 modifying op. A stock crash was not observed in ~8 full forced-umount cycles under working unpriv trigger load (run.stock2.log) -- expected for a few-ns window without KASAN; the primitive is a silent +/-2 or bit-0 atomic write onto freed M_TEMP memory whose address is reused by later mounts (cross-mount corruption observed). Fix (mnt_token shared/exclusive interlock + NULL guards, fix.diff) validated on-guest: identical procedure yields parked=3 -> FREE with NO HIT and no NULL-deref panic.", "exploit_chain": "unprivileged user churns creates/writes on a hammer2 mount (drives hammer2_pfs_memory_wait -> trigger_syncer*); privileged side runs umount -f concurrently; any trigger caller preempted between its mnt_syncer_ctx load and the atomic op when vn_syncer_thr_stop frees the ctx performs an atomic add +/-2 (or set bit 0) on freed kernel heap (M_TEMP objcache; addresses observed reused by subsequent mounts) -> latent heap corruption / cross-mount syncer_trigger corruption; the caller's struct mount pointer is itself unreferenced, so mp->mnt_syncer_ctx can also be a UAF read of freed M_MOUNT.", "evidence": [ "panic.txt: 'DF2832: stop: parked=6' -> 'thr_stop FREE ctx=...' -> interleaved 'DF2832: HIT stale ctx=0xn...' + panic; 'Stopped at lwkt_gettoken+0x64'", "run.fixvalidation.log: fix build 'stop: parked=3' -> 'thr_stop FREE' with NO HIT line and no lwkt_gettoken panic", "run.stock2.log: stock kernel, working unpriv storm (3 dirty_writer + 6 churn_open), umount -f cycles, no panic", "witness_kernel_baseline.c / witness_kernel_fixed.c: the two in-guest instrumented kernels (park at the exact stock boundary; debug.df2832_stop invokes the real vn_syncer_thr_stop)", "fix.diff: repo-side mnt_token interlock + NULL guards" ], "kernel_refs": [ "sys/kern/vfs_sync.c:561-572", "sys/kern/vfs_sync.c:579-588", "sys/kern/vfs_sync.c:590-598", "sys/kern/vfs_sync.c:603-614", "sys/kern/vfs_sync.c:330-358", "sys/kern/vfs_vfsops.c:124-138", "sys/kern/vfs_syscalls.c:1107-1117", "sys/kern/vfs_vnops.c:474-485", "sys/vfs/hammer2/hammer2_vfsops.c:2925-3007", "sys/vfs/hammer2/hammer2_flush.c:272" ], "poc_changes": "Pass-2 new finding (no seed). Built from scratch; notable iterations: /tmp tmpfs denies exec for nobody (binaries moved to /usr/local); csh su -c redirections need csh syntax; guest sysctl rejects hex 64-bit (dropped the pointer sysctl, auto-scoped witness off the root fs via rootvnode->v_mount); tsleep-park wedged the box when unscoped (root-fs trigger ops parked); full-umount variant cannot complete under storm (hammer2 dirty-chain backpressure oscillation in VFS_SYNC(MNT_WAIT)) so the deterministic witness sysctl invokes the real vn_syncer_thr_stop directly; fix-validation park switched to busy-spin because tsleep drops lwkt tokens and no longer emulates preemption.", "attempts": 8, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 11400, "guest_dirty": 0, "build_cmd": "cc -O2 -o churn_open churn_open.c && cc -O2 -o dirty_writer dirty_writer.c (guest; witness kernels: cp witness_kernel_*.c /usr/src/sys/kern/vfs_sync.c && make -j6 nativekernel && make installkernel)", "run_cmd": "sh race_stock2.sh 10 (stock); witness: mount_hammer2 /dev/vn0 /mnt/t; 6x 'su -m nobody -c churn_open /mnt/t N'; wait debug.df2832_parked>0; sysctl debug.df2832_stop=1; dmesg | grep DF2832", "code_hash": "712d458af68462729e12fe2501ec1ca7a6ef1af7e76af6bf5ac5c1627870e0d1", "notes": "Honesty: (1) stock kernel did NOT crash in ~8 forced-umount cycles under working unpriv trigger load (ns window vs s-long umount, no KASAN); an earlier 40-cycle stock run had a defective harness (unpriv binaries never executed - /tmp exec denial) and only exercised the teardown side; (2) the witness park is a faithful preemption emulation at the exact load->RMW boundary for the stock code (which holds no lock there); teardown used is the REAL vn_syncer_thr_stop invoked deterministically (the umount(2) path reaches the same function at vfs_vfsops.c:135); (3) baseline-run panic at lwkt_gettoken+0x64 is the NULL-ctx facet - same API defect, stock-unreachable through normal unmount ordering but reachable via any live-mount thr_stop; (4) fix-validation run's 'hashdestroy: hash not empty' panic is an artifact of tearing down a live mount (bypasses VFS_UNMOUNT's vflush) and is orthogonal to the fix; (5) side observation: unpriv dirty-chain pressure can wedge umount -f indefinitely in VFS_SYNC(MNT_WAIT) (hammer2 backpressure oscillation) - potential separate livelock issue; (6) ctx addresses are objcache-reused across mounts, so late stale RMWs corrupt the NEXT mount's syncer_trigger.", "recommended_fix": "Interlock the syncer-trigger API against vn_syncer_thr_stop: hold mnt_token shared across the mnt_syncer_ctx load + RMW in trigger_syncer/trigger_syncer_start/trigger_syncer_stop/speedup_syncer, and NULL-publish + hashdestroy + kfree under mnt_token exclusive in vn_syncer_thr_stop; add NULL-ctx guards in vn_syncer_add/vn_syncer_remove/vsetisdirty/vsetobjdirty (see fix.diff).", "fix_status": "fixed", "fix_kernel_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #5: Wed Sep 2 02:35:06 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64 (witness+fix build, in-guest only)", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "Baseline witness (stock semantics): parked=6 in-window callers -> real vn_syncer_thr_stop free -> 'DF2832: HIT stale ctx' (window proven) + lwkt_gettoken NULL-ctx panic. Patched (witness+fix.diff, busy-spin park which holds mnt_token shared like a preempted fixed thread): identical procedure -> parked=3 -> FREE with NO HIT and no NULL-deref panic; only the orthogonal 'hashdestroy: hash not empty' artifact KASSERT (teardown-of-live-mount, fires regardless of the fix). The load->RMW vs free interleaving is closed by the mnt_token interlock; the NULL-ctx facet is closed by the guards. Guest left on stock kernel #0 after reset.", "fix_evidence": [ "run.fixvalidation.log: 'DF2832: stop: parked=3, tearing down syncer of mp=0xfffff8008fa6b000' -> 'DF2832: thr_stop FREE ctx=0xfffff8011750e9a0' with no HIT line", "panic.txt (baseline): HIT + lwkt_gettoken+0x64 panic" ] } |