MFS bio queue shared between BGL-holding producers and a BGL-free consumer: unsynchronized cross-CPU TAILQ corruption (kernel panic; latent heap corruption on non-INVARIANTS builds) from unprivileged file I/O on a mounted MFS
Summary
mfsstrategy() enqueues bios into mfsp->bio_queue under the BGL (mfs_ops lacks D_MPSAFE -> dev_dstrategy get_mplock, kern_device.c:384-388) while mfs_start() dequeues the same lockless TAILQ under only crit_enter() - a local-CPU critical section - because mfs_mount() sets MNTK_ST_MPSAFE (:401) so vfs_start() takes no BGL. Producer and consumer share no common lock: on SMP the insert races the remove, dropping bios (unkillable biowait hang) and leaving stale tqh_last pointers whose next insert writes a kernel pointer into recycled buffer-cache memory. Any unpriv user performing file I/O on a mounted MFS (classic /tmp-as-MFS deployment) races the single I/O server thread across CPUs. VERIFIED twice on stock INVARIANTS guest: 'panic: Bad link elm prev->next != elm' from mfs_start after ~3 min of 6-writer stress, and 'panic: Bad tailq NEXT != NULL' from mfsstrategy (ufs_strategy stack) in <90s. Production kernels: reliable unpriv DoS + silent heap corruption (uncontrolled pointer store into recycled struct buf). uid=0 push rigorously blocked (stray write lands in recycled buffer-cache struct-buf memory, fixed offset, kernel-managed reuse, value not attacker-chosen; no allocator path exposes that slot to credential-bearing objects). Fix validated on two patched in-guest kernels (per-mfsnode spinlock + producer ENXIO self-complete + final drain): 11+ min identical stress zero panics, normal lifecycle intact.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3170 Β· 14 files| File | Type | Description | Size | |
|---|---|---|---|---|
| mfsrace.c | β | 2.8 KB | view raw | |
| build.sh | β | 119 B | view raw | |
| run.sh | β | 255 B | view raw | |
| run.log | β | 1.9 KB | view raw | |
| run.2.log | β | 1.9 KB | view raw | |
| panic.txt | β | 541 B | view raw | |
| panic_run2.txt | β | 812 B | view raw | |
| run2_panic_serial.log | β | 13.6 KB | view raw | |
| env.txt | β | 517 B | view raw | |
| VERDICT.md | β | 6.0 KB | β raw | |
| fix.diff | β | 3.6 KB | view raw | |
| fix_build.log | β | 1.3 KB | view raw | |
| fix_run.log | β | 2.0 KB | view raw | |
| verdict.json | β | 4.5 KB | view raw |
DF-3170 β VERDICT
Status: reproduced (panic). Impact: panic (kernel memory corruption primitive: cross-CPU corruption of an in-kernel TAILQ). Confidence: certain. Two independent stock-kernel manifestations: consumer-side "Bad link elm" from mfs_start (~3 min of stress) and producer-side "Bad tailq NEXT" from mfsstrategy (<90 s of stress) β see run.log, run.2.log, panic.txt, panic_run2.txt.
Guest
DragonFly dfbsd 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026
(X86_64_GENERIC, INVARIANTS), QEMU/KVM, 6 vCPUs (hw.ncpu: 6),
stock snapshot. Booted fresh via vm.sh reset with-src before the run.
How it was reproduced
- root:
mount_mfs -s 32768 swap /mnt/mfsβ server pid 899 parked inmfsidl(the mfs_start loop), device/dev/mfs899, mount live. - unprivileged user
maxx: 6 Γ/tmp/mfsrace /mnt/mfs 2β six processes each hammering 2 files with 512-byte O_SYNC pwrite/pread loops. - Within ~3 minutes the guest panicked:
panic: Bad link elm 0xfffff8004f593d48 prev->next != elm
cpuid = 5
Trace beginning at frame 0xfffff80117f12668
mfs_start() at mfs_start+0x499 0xffffffff80913849
mfs_start() at mfs_start+0x499 0xffffffff80913849
vfs_start() at vfs_start+0x3b 0xffffffff8070d55b
sys_mount() at sys_mount+0x52f 0xffffffff80700ccf
syscall2() at syscall2+0x11e 0xffffffff80bd6a0e
Debugger("panic")
(panic.txt, boot serial log.) ssh died; vm.sh status β down.
Why this proves the root cause
- The panicking frame is
mfs_startβ the consumer ofmfsp->bio_queueβ insidebioq_takefirst/bioq_remove(sys/sys/buf2.h:148-175), hittingTAILQ_REMOVE's INVARIANTS checkprev->next != elm(sys/sys/queue.h:505-519 family). - The only writers of that TAILQ are
bioq_insert_tailfrommfsstrategy(mfs_vfsops.c:187) and the dequeue inmfs_start(mfs_vfsops.c:455). The producer runs under the BGL (mfs_opshas noD_MPSAFE; kern/kern_device.c:121,384-388) while the consumer deliberately does not (MNTK_ST_MPSAFEset at mfs_vfsops.c:401 βVFS_MPLOCK_FLAGno-ops in vfs_start, sys/sys/mount.h:408-417). No other lock exists. Therefore the corrupted linkage can only come from a producer racing the consumer across CPUs β exactly the interleaving predicted. - cpuid=5 vs the server thread: 6-way SMP stress; every writer is unprivileged.
Exploit-chain assessment (why impact is filed as panic, not uid0)
On the INVARIANTS validation kernel every queue inconsistency traps immediately, so the observed result is a panic. On a production kernel the same interleavings give:
- Lost bio β dropped
TAILQ_INSERTupdate: the bio never completes; the issuing thread blocks uninterruptibly inbiowaitforever (local DoS, very reliable to cause, not steerable). - Stale
tqh_lastpointer store β after a lost-update removal,bio_queue.tqh_lastcan point at thebio_act.tqe_nextfield of a bio that has beenbiodoned and whosestruct bufreturned to the buffer cache. The nextbioq_insert_tailthen executes*(head)->tqh_last = elm, writing a kernel heap pointer (address of the new bio's buf β not attacker-chosen) into recycled kernel memory.
Pushing (2) to uid=0 requires all of: winning the few-instruction race on
demand; the recycled victim struct buf being reused as an object whose
corruption at bio_act's fixed offset yields a privilege-relevant
primitive; and the value written (a buf address) being useful at that
slot. Buffer-cache bufs are recycled kernel-internally (bufdaemon) into the
same struct buf type β the stray store lands inside another buffer
descriptor, corrupting buffer linkage rather than credentials; there is no
allocator path that hands that slab slot to a ucred/proc/vtable object
while the dangling tqh_last still references it, and the write value is
never controlled. Documented blocker β no credible uid0 chain; the
deliverable stands at reliable unprivileged kernel panic (+ latent
corruption on non-INVARIANTS builds).
Negative space (checked and killed during pass 2)
copyinstr(args.fspec,...)unchecked +sizeused inbzero(f_mntfromname+size, MNAMELEN-size)(mfs_vfsops.c:376-380): NOT a bug on pc64 βstd_copyinstrstores*lencopiedon ALL return paths, including the fault path (sys/platform/pc64/x86_64/support.s:705-715), and the stored value is always β€ MNAMELEN-1.vget(devvp)overwriting the ENOENT guard whendevvp == NULL(mfs_vfsops.c:350-354): unreachable βnlookupwith flags 0 returns ENOENT for negative-cache entries (kern/vfs_nlookup.c:891,905), so theerror == 0branch always has a non-NULLnc_vp. Hardening wart only.bp->b_bcount = mfsp->mfs_size - bofftruncation (mfs_vfsops.c:178): clip only executes when the remainder is smaller than the request, and requests are bounded by MAXPHYS/MAXBSIZE β no int truncation possible.- devvp double-vrele on
ffs_mountfsfailure: killed βffs_mountfsnever vreles devvp on its error paths (ffs_vfsops.c:774-787), so mfs_vfsops.c:406 is the correct single drop. - crafted
fs->fs_size>mfs_size: no OOB β every access goes throughmfsstrategyclip/EINVAL (mfs_vfsops.c:163-179) andmfs_doio's copyin/copyout, both fault-bounded. - MNT_UPDATE path / EMOUNTEXIT plumbing (vfs_vfsops.c:115-116): correct; mfs_start returns only after dounmount() completed.
Fix validation
See fix_build.log / fix_run.log and the fix_* fields in verdict.json.
Combined single-fix kernel (this fix + DF-3171's guard, which touches the
same function) built in-guest with make nativekernel; the exact PoC
stress and the DF-3171 kill sequence were re-run on it: no panic, mount
I/O continues, unmount clean. Baseline (stock kernel) panicked in both
runs.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedBaseline stock kernel panicked twice under the writer stress (consumer- and producer-side TAILQ INVARIANTS checks). Patched kernel (build #1 and rebuilt #2) survived 11+ minutes of the identical stress at equal or higher op rates plus ENOSPC churn and a forced-devfs-unmount kill construction, with all writers progressing monotonically and all three MFS servers alive; normal mount/IO/umount lifecycle intact (server exits via EMOUNTEXIT, /dev/mfsN removed). No panic, no wedge, no lost bios.
findings/poc/DF-3170/fix_build.log, findings/poc/DF-3170/fix_run.log, findings/poc/DF-3171/fix_run.log [2]
Confirmed kernel references
Detail
Exploit chain
unpriv writers -> O_SYNC pwrite/pread on mounted MFS -> ufs_strategy -> dev_dstrategy(BGL) -> mfsstrategy TAILQ_INSERT_TAIL racing mfs_start's bioq_takefirst on another CPU -> TAILQ corruption (panic on INVARIANTS). uid0 chain rigorously blocked: the corruption primitive is an uncontrolled kernel-pointer store through a dangling tqh_last that lands inside recycled buffer-cache struct-buf memory (kernel-managed reuse, fixed offset, value = bio/ buf address, not attacker-chosen); no allocator path exposes that slot to credential-bearing objects, so no credible pivot to ucred/proc was found. Primary impact: reliable unprivileged kernel panic/DoS + latent heap corruption on production kernels.
Evidence (decisive lines)
['panic.txt (Bad link elm, mfs_start stack, cpuid 5)', 'panic_run2.txt + run2_panic_serial.log (Bad tailq NEXT, mfsstrategy/ufs_strategy stack, cpuid 2)', 'run.log (full baseline sequence)', 'fix_run.log (patched-kernel stress, no panic)', 'VERDICT.md']
PoC changes
PoC authored fresh for this finding (mfsrace.c: 6 processes x 2 files of 512-byte O_SYNC pwrite/pread at rotating offsets). Guest-side: launch with nohup + full redirects so writers survive ssh session close (the naive '2 > file' form leaks stderr onto the session and SIGHUPs writers); keep total file set <= 12MB on a 16MB MFS so writers never ENOSPC.
Verified recommended fix
Serialize mfsp->bio_queue with a per-mfsnode spinlock in mfsstrategy/mfsclose/mfs_start (producers self-complete ENXIO when inactive; final drain in mfs_start), replacing the BGL/crit non-pairing.
Verdict
Unprivileged concurrent file I/O on a mounted MFS corrupts the mfs bio queue: mfsstrategy enqueues under the BGL while mfs_start dequeues BGL-free under only a local critical section (MNTK_ST_MPSAFE is set), so producer and consumer share a lockless TAILQ with no common lock. Two independent stock-kernel panics reproduced from unprivileged writers (consumer-side 'Bad link elm prev->next != elm' in mfs_start after ~3 min; producer-side 'Bad tailq NEXT(...->tqh_last) != NULL' in mfsstrategy under 90 s). On non-INVARIANTS kernels the same interleavings are a dropped bio (unkillable biowait hang) and a stale-tqh_last pointer store into recycled kernel memory. Spinlock fix validated on two patched in-guest kernels: 11+ min of baseline-equivalent stress (4-6M iterations per writer, two mounts, ENOSPC churn) with zero panics and normal mount/IO/umount lifecycle intact.
No comments yet.