DF-2832 — syncer-trigger API UAF vs vn_syncer_thr_stop teardown

File: sys/kern/vfs_sync.c Class: CWE-362 / CWE-416 (race → use-after-free write, kernel heap) Severity: Medium (memcorrupt bucket) · Confidence: certain (defect), window demonstrated on-guest (witness); stock crash not observed (expected).

What the bug is

trigger_syncer(), trigger_syncer_start(), trigger_syncer_stop() and speedup_syncer() (sys/kern/vfs_sync.c:561-614) load mp->mnt_syncer_ctx lock-free and then operate on the struct syncer_ctx (atomic add ±2 / set-bit-0 / wakeup). vn_syncer_thr_stop() (vfs_sync.c:330-358) publishes mp->mnt_syncer_ctx = NULL and then hashdestroy()+kfree()s the ctx with no synchronization against those readers. A thread preempted between the load and the RMW completes an atomic write on freed heap memory (M_TEMP). The callers' mp pointer itself is also unreferenced (dounmount frees the struct mount at vfs_syscalls.c:1107-1117), so mp->mnt_syncer_ctx can be a read of freed M_MOUNT memory.

Reachability (unprivileged side): any process performing hammer2 modifying ops under dirty-chain pressure: open(O_CREAT|O_WRONLY) → ncp_writechk (vfs_vnops.c:481) → VFS_MODIFYING → hammer2_vfs_modifying (hammer2_vfsops.c:2925) → hammer2_pfs_memory_wait (hammer2_vfsops.c:2938) → trigger_syncer/trigger_syncer_start (hammer2_vfsops.c:2962/2977). hammer2_flush.c:272 (speedup_syncer) is a kernel-thread caller. The privileged side is umount -f → vfs_unmount → vn_syncer_thr_stop (vfs_vfsops.c:135).

Notable: a thread stalled in hammer2_pfs_memory_wait during open() holds no fd on the mount, so dounmount's process-kill scan (process_uses_mount, vfs_syscalls.c:727-757) does not match it — the trigger storm runs all the way through the forced-unmount teardown.

What is in this pack

Reproduce (summary)

1. Stock race (no crash expected; window is ns vs umount's ms): sh race_stock2.sh 10 as root in the guest.
2. Witness proof: build witness_kernel_baseline.c as the kernel (cp over /usr/src/sys/kern/vfs_sync.c, make nativekernel && make installkernel, reboot), then mount a vn-backed hammer2 fs, run 6× churn_open as nobody, wait for debug.df2832_parked > 0, then sysctl debug.df2832_stop=1. Console shows stop: parked=N → thr_stop FREE ctx=… → DF2832: HIT stale ctx=… (+ the artifact panic below).
3. Fix validation: build witness_kernel_fixed.c, same procedure → parked=N → FREE, no HIT.

DF-2832 — VERDICT

Bottom line: the defect is real and its exact interleaving was demonstrated on a running kernel; a stock-kernel crash was not observed (and is not expected to be observable without KASAN, because the primitive is a bounded atomic write (±2 / bit-0) onto freed heap, usually silently absorbed). The fix closes the window and was validated on-guest.

1. Code-level proof (no run needed)

• trigger_syncer*()/speedup_syncer() read mp->mnt_syncer_ctx and RMW the ctx with no lock, no refcount, no mount hold — vfs_sync.c:570-571, 584-586, 595-596, 608-611.
• vn_syncer_thr_stop() NULLs the pointer under sc_token (which the trigger family never takes) and then hashdestroy+kfrees the ctx — vfs_sync.c:353-357. hashdestroy's only protection is a KASSERT on the lists, nothing protects the syncer_trigger field readers.
• The caller's struct mount * is also unreferenced; dounmount can free it (vfs_syscalls.c:1107-1117) while a hammer2 frontend thread is inside the trigger functions.
• The other side of the same API defect: after the NULL publish, any lock-free vn_syncer_add/vsetisdirty (reassignbuf — every dirty buffer) does lwkt_gettoken(&ctx->sc_token) with ctx == NULL → guaranteed page fault. In the stock ordering this facet is unreachable (VFS_UNMOUNT's vflush drains all vnodes before thr_stop runs — verified: vgone_vxlocked force-dequeues at vfs_subr.c:1573-1580, and dounmount panics on dangling vnodes at vfs_syscalls.c:1085-1086); it becomes reachable the moment any code path stops the syncer on a live mount (demonstrated below; the failed-mount thr_stop paths at vfs_syscalls.c:458 / vfs_conf.c:369,508 are the speculative stock candidates).

2. On-guest demonstration (witness kernels, guest-only instrumentation)

The witness kernel inserts a park at the exact stock instruction boundary — after ctx = mp->mnt_syncer_ctx and before the atomic RMW — which is precisely where a preemption lands on the stock kernel. A park is a faithful preemption emulation for the stock code because the stock code holds no lock there. The teardown is the real vn_syncer_thr_stop() (invoked deterministically via a debug sysctl once parks are observed, because the full umount -f path cannot complete while the unpriv storm saturates hammer2's dirty-chain backpressure — see §4). Load side is genuinely unprivileged: nobody running churn_open (stall reached via open(O_CREAT) → ncp_writechk → VFS_MODIFYING → hammer2_pfs_memory_wait).

Decisive run (panic.txt):

DF2832: stop: parked=6, tearing down syncer of mp=0xfffff8008fa6b000
DF2832: thr_stop FREE ctx=0xfffff801174ee9a0 mp=0xfffff8008fa6b000
pan[D]iF…2:8 …HIT stale ctx=0xn f…   <- kprintf streams interleaved
Stopped at  lwkt_gettoken+0x64:  movq (%r12),%rax      <- db>

• parked=6: six unprivileged trigger callers were inside the window.
• FREE: the real vn_syncer_thr_stop destroyed the ctx under them.
• HIT stale ctx=0x…: a parked caller observed the teardown — on the stock code it would have executed atomic_fetchadd_int(&ctx->syncer_trigger, 2) on freed memory.
• The lwkt_gettoken panic is the NULL-ctx facet (a churner's vn_syncer_add after teardown of the still-live mount) — same API defect, second facet, artifact of the deterministic teardown bypassing VFS_UNMOUNT ordering (not stock-reachable; see §1).

Also observed across cycles: the freed ctx address is reused by the next mount (0xfffff80117b51ee0/…ee0/…f40 repeatedly), so a stale RMW that lands late corrupts the next mount's syncer_trigger — cross-mount corruption, silent.

3. Fix validation (fix.diff)

fix.diff (against the repo sys/, never applied there): 1. trigger_syncer/trigger_syncer_start/trigger_syncer_stop/ speedup_syncer hold mnt_token shared across load+RMW; 2. vn_syncer_thr_stop NULLs+frees under mnt_token exclusive; 3. NULL guards in vn_syncer_add/vn_syncer_remove/vsetisdirty/ vsetobjdirty (defense-in-depth; also what stops the §2 NULL-panic facet).

Validation build = witness + fix, with the park switched to a busy-spin (a tsleep would drop the lwkt token and no longer emulate preemption; real preemption holds tokens). Identical procedure:

DF2832: stop: parked=3, tearing down syncer of mp=0xfffff8008fa6b000
DF2832: thr_stop FREE ctx=0xfffff8011750e9a0 mp=0xfffff8008fa6b000
panic: hashdestroy: hash not empty        <- artifact, see below

• No HIT: with the interlock, the teardown cannot slip between the load and the RMW — the exclusive acquire blocks until every in-window caller completes. The window demonstrated in §2 is closed.
• No lwkt_gettoken NULL-deref panic (guards work; execution proceeds until the teardown-of-a-live-mount hits hashdestroy's "hash not empty" KASSERT — an artifact of bypassing VFS_UNMOUNT's vflush that is orthogonal to the fix and would fire identically on the unfixed kernel if the NULL-deref hadn't fired first).

4. Honesty notes

• Stock full-umount race: 40 iterations early (defective harness — /tmp denies exec for nobody, so the unpriv binaries never ran; only the teardown side was exercised) + ~8 iterations after the fix (run.stock2.log, working unpriv storm: 3×dirty_writer + 6×churn_open, umount -f cycles) — no crash. Expected: the window is a few ns vs. an umount taking seconds; without KASAN the ±2 write is invisible.
• Why the deterministic sysctl teardown: with the storm saturating vfs.hammer2.limit_dirty_chains, umount -f cannot get through VFS_SYNC(mp, MNT_WAIT) (vfs_syscalls.c:1002) — hammer2's backpressure oscillates (churners stall → chains drain → churners resume) forever. That interaction (unpriv dirty-pressure wedging forced unmount in VFS_SYNC) may deserve its own look; it is a livelock, not corruption.
• Witness parks were scoped to never fire on the root filesystem (an unscoped first attempt wedged the guest; instrumentation artifact).
• stat_rush_requests non-atomic increment (vfs_sync.c:569) observed while reading — cosmetic, not filed.