Unsynchronized UFS quotaon/quotaoff mount-state machine: concurrent Q_QUOTAOFF double-executes the teardown β double vn_close(qvp) (vnode opencount/refcount corruption, UAF), crfree(NOCRED) NULL-deref, and dqsync-against-closed-quota-file panic
Summary
ufs_quotaoff() never tests QTF_CLOSING (set at :505, read only by ufs_dqget at :762) and shares no lock with ufs_quotaon() or another ufs_quotaoff(); vfs_busy() in ufs_quotactl (ufs_vfsops.c:141) is a shared unmount pin, not mutual exclusion. Two quota-admin-capable threads that both pass the um_quotas[type]==NULLVP check at :503 execute the whole teardown N times: double vn_close(qvp) at :519 -> VOP_STDCLOSE BAD OPENCOUNT panic / vnode refcount desync -> premature recycle while dquot cache still references it (UAF); the loser's crfree(um_cred[type]) at :521 after the winner stored NOCRED at :522 -> NULL write-deref in crfree; the loser's scan-side dqrele->dqsync after :520 cleared um_quotas[type] -> panic('dqsync: file') at :909 or VOP_WRITE on a closed quota file; and the loser's um_quotas[type]=NULLVP store can clobber a concurrent re-quotaon's live pointers. Distinct root cause from DF-0847 (global dquot hash) and DF-0846 (sequential leak): the mount-level quota state machine. Both racing threads must pass SYSCAP_NOQUOTA_WR - not an unpriv escalation; the realistic boundary is a capability-delegated quota manager (jail/helper holding only the quota capability) turning narrow privilege into a vnode UAF, plus trivially repeatable panics. Requires 'options QUOTA' (absent from stock X86_64_GENERIC). VERIFIED on a QUOTA-rebuilt INVARIANTS guest: qt_race.c (2 Q_QUOTAON + 4 Q_QUOTAOFF threads + unpriv churn) -> baseline panic <=45s 'VOP_STDCLOSE: BAD OPENCOUNT' (the double close) + instrumented run trapped at crfree+0xc fault 0x0 (the NOCRED crfree). Fix validated (per-mount um_quota_lock serializing the state machine + lock-order fix): identical PoC survived 300s (1.5M on + 17.8M off) + 120s confirmation, zero panics.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3120 Β· 13 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 2.6 KB | β raw | |
| VERDICT.md | β | 5.6 KB | β raw | |
| qt_race.c | β | 4.8 KB | view raw | |
| build.sh | β | 211 B | view raw | |
| run.sh | β | 185 B | view raw | |
| build.log | β | 611 B | view raw | |
| run.log | β | 350 B | view raw | |
| panic.txt | β | 2.2 KB | view raw | |
| panic.diag.txt | β | 2.1 KB | view raw | |
| env.txt | β | 249 B | view raw | |
| fix.diff | β | 3.5 KB | view raw | |
| manifest.json | β | 2.6 KB | view raw | |
| verdict.json | β | 5.5 KB | view raw |
DF-3120 β unsynchronized UFS quotaon/quotaoff state machine
Unsynchronized ufs_quotaon() / ufs_quotaoff() double-execution race in
sys/vfs/ufs/ufs_quota.c. Two threads holding SYSCAP_NOQUOTA_WR (any
quotactl-capable principal, e.g. root or a delegated quota manager) that both
pass the um_quotas[type] == NULLVP check at ufs_quota.c:503 run the whole
quotaoff teardown twice:
- double
vn_close(qvp)at:519β vnode refcount underflow / UAF (INVARIANTS:KKASSERTinvrele,sys/kern/vfs_lock.c:354) - loser's
crfree(ump->um_cred[type])at:521after the winner storedNOCRED((struct ucred *)0,sys/sys/ucred.h:82) at:522β NULL pointer deref incrfree(sys/kern/kern_prot.c) - loser's scan-side
dqreleβdqsyncafterum_quotas[type]cleared at:520βpanic("dqsync: file")(ufs_quota.c:909) orVOP_WRITEon a closed quota file - loser's
um_quotas[type] = NULLVPstore can clobber a concurrent re-quotaon's live quota-file pointer (leaked vp + laterdqsync: file)
There is no lock: QTF_CLOSING is set at :505 but read only by
ufs_dqget (:762) β ufs_quotaoff itself never tests it, and
vfs_busy() in ufs_quotactl (sys/vfs/ufs/ufs_vfsops.c:141) is a shared
unmount pin, not mutual exclusion.
PoC
qt_race.c β 6 quotactl threads (2ΓQ_QUOTAON rotating 8 quota files,
4ΓQ_QUOTAOFF, each using a distinct control path), plus unprivileged
children (uid 65534) doing Q_SYNC/Q_GETQUOTA and file churn so dquots exist
and stay dirty. The distinct control paths matter: when every thread
nlookup()ed the same control file, the shared namecache entry serialized
the whole syscall and the race could not manifest (measured: 1.5M quotactls,
zero overlaps, instrumented kernel). With distinct paths the overlap happens
within seconds.
Build (in guest, as root)
cc -O -pthread -o /root/qt_race /root/qt_race.c
Guest setup (stock kernel has no options QUOTA β a QUOTA kernel
must be built first, see VERDICT.md "not_testable on stock config" note)
dd if=/dev/zero of=/qt.img bs=1m count=64 vnconfig -c /dev/vn0 /qt.img newfs -m 0 /dev/vn0 mkdir -p /qtmnt && mount -t ufs /dev/vn0 /qtmnt (PoC creates /qtmnt/ctrlN and /qtmnt/q0..q7 itself)
Run
/root/qt_race 100
Expected
Kernel panic on the serial console within ~10-100 s:
Fatal trap 12: page fault while in kernel mode fault virtual address = 0x0 Stopped at crfree+0xc: cmpq $0,(%rdi)
(the crfree(NOCRED) NULL-deref). SSH dies / vm.sh status => down.
With the INVARIANTS vrele assert or a dq-holding interleaving you can also
see panic("dqsync: file") / vrele KKASSERT variants.
DF-3120 VERDICT β unsynchronized UFS quotaon/quotaoff state machine
Bottom line
REPRODUCED (kernel panic, race double-execution of ufs_quotaoff)
on a DragonFly 6.5-DEVELOPMENT x86_64 kernel built with options QUOTA
(stock X86_64_GENERIC has no options QUOTA, so the surface is compiled
out of the default kernel β ufs_quotactl returns EOPNOTSUPP,
sys/vfs/ufs/ufs_vfsops.c:77-78). With the QUOTA option the race
panics the kernel within ~45 s of hammering; the fix serializes
quotaon/quotaoff per-mount and survives the identical hammer.
The corruption is gated behind caps_priv_check(SYSCAP_NOQUOTA_WR)
(both racing threads must be quota-admin-capable, normally root), so this
is a Medium "privileged-admin β kernel memory corruption/panic" finding,
not an unprivileged escalation. There is no unprivileged trigger path:
every entry into ufs_quotaon/ufs_quotaoff is via
quotactl(2) (privilege-checked) or unmount (ffs_flushfiles,
ffs_vfsops.c:891, serialized against quotactl by MNTK_UNMOUNT /
vfs_busy). The realistic impact ceiling is a capability-delegated quota
manager (jail/helper holding only SYSCAP_NOQUOTA_WR) converting its
narrow privilege into kernel UAF β host takeover.
Root cause (path:line)
sys/vfs/ufs/ufs_quota.c:495-531βufs_quotaoff(): entry testif ((qvp = ump->um_quotas[type]) == NULLVP) return (0);at:503,QTF_CLOSINGset at:505but never tested here β the only reader isufs_dqgetat:762. Teardown:ufs_dqflush(qvp):517,vn_close(qvp):519,um_quotas[type] = NULLVP:520,crfree(um_cred[type]):521,um_cred[type] = NOCRED:522.sys/vfs/ufs/ufs_quota.c:412-475βufs_quotaon(): state swapif (*vpp != vp) ufs_quotaoff(mp, type)at:437-438,*vpp = vp:442,um_cred[type] = crhold(cred):448β same absence of locks.sys/vfs/ufs/ufs_vfsops.c:141βvfs_busy(mp, LK_NOWAIT)is a shared pin against unmount, not mutual exclusion between quotactls.- Consequence: N threads passing
:503together execute the teardown N times β doublevn_close(vnode refcount/opencount underflow),crfree(NOCRED)withNOCRED == (struct ucred *)0(sys/sys/ucred.h:82) β NULL write-deref incrfree(sys/kern/kern_prot.c,cr->cr_refat +0x10), stale-taildqreleβdqsyncseeingum_quotas[type]cleared βpanic("dqsync: file")ufs_quota.c:909/VOP_WRITEon a closed quota file, and clobbering of a concurrent re-quotaon's liveum_quotas[type]/um_cred[type]pointers (leak + later panic).
Reproduction timeline (all on dfbsd-qemu guest, 6 vCPU)
- Stock kernel:
quotactl(Q_QUOTAON)=EOPNOTSUPP(no QUOTA option inX86_64_GENERIC). BuiltX86_64_QUOTA(cp X86_64_GENERIC + options QUOTA) withmake nativekernel. - Instrumented kernel (diag counters only, no semantic change):
PoC with a shared control path never overlapped (1.5M quotactls,
0 window entries β the shared namecache entry of the single ctrl
file serialized the syscalls by accident). PoC v3 with distinct
control paths per thread (
/qtmnt/ctrl0..5): panic in <60 s βpanic.diag.txt:DF3120-DIAG: race window entered (in-flight=1)immediately followed byFatal trap 12 ... fault virtual address = 0x0,Stopped at crfree+0xc: cmpq $0,(%rdi)β the predictedcrfree(NOCRED)NULL-deref atufs_quota.c:521. - Clean baseline kernel (pristine
ufs_quota.c, onlyoptions QUOTAadded, INVARIANTS on): identical PoC panicked in β€45 s βpanic.txt:panic: VOP_STDCLOSE: BAD OPENCOUNTwith stackvop_stdclose β vop_close β vn_close β ufs_quotaoff β ufs_quotactlβ the double-vn_close(qvp)manifestation of the same race (ufs_quota.c:519executed twice). - Fix v1 (
fix.diff,um_quota_lockaround quotaon/quotaoff): no panic, but my own validation caught a fix deadlock β quotaon originally tookum_quota_lockwhile still holding the quota-file vp lock returned byvn_open(beforevn_unlock(vp)), inverting against quotaoff'sum_quota_lock β vget()order. All quotactl threads stuck inDstate, wchanncplk. Fixed by unlocking the vp before taking the lock (documented infix.diffcomment). - Fix v2 (final
fix.diff): identical PoC, 300 s + repeat runs: guest up, no panic, mount healthy (see run logs). Baseline panicked β€45 s; patched survived every run β behavior change demonstrated.
Exploitability assessment
Memory-corruption class, but both racing threads must pass
SYSCAP_NOQUOTA_WR; no unprivileged entry exists. Escalation ceiling:
a delegated quota-capable process (not uid 0) double-closing the quota
vnode β refcount desync β premature vnode recycle while
um_quotas[type]/dquot cache still reference it β UAF on later
dqsync I/O. Not pursued to uid=0 because the entry gate already
requires the privilege the escalation would grant (root-vs-root has no
boundary to cross); the panic is the honest demonstrated impact.
Fix
fix.diff β per-mount um_quota_lock (lockmgr) serializing
ufs_quotaon/ufs_quotaoff (ufs_quotaoff wraps a
ufs_quotaoff_locked body so quotaon's internal swap-call does not
self-deadlock); lockinit in ffs_mountfs; vn_unlock(vp) before
taking the lock in quotaon (deadlock note). Validated on-guest.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedBaseline panicked <=45s ('VOP_STDCLOSE: BAD OPENCOUNT' via ufs_quotaoff->vn_close); patched kernel survived the identical PoC for 300s (1.5M Q_QUOTAON + 17.8M Q_QUOTAOFF) plus a 120s confirmation run, exit 0, zero panics, mount healthy. Bad behavior gone.
['findings/poc/DF-3120/panic.txt (baseline panic)', 'findings/poc/DF-3120/run.log (patched SURVIVED, RC=0, x2 runs)', 'findings/poc/DF-3120/fix.diff (validated, patch -p1 byte-identical to tested source)']
Confirmed kernel references
Detail
Evidence (decisive lines)
["findings/poc/DF-3120/panic.txt β baseline panic: 'panic: VOP_STDCLOSE: BAD OPENCOUNT', stack vop_stdclose<-vop_close<-vn_close<-ufs_quotaoff<-ufs_quotactl", "findings/poc/DF-3120/panic.diag.txt β instrumented run: 'DF3120-DIAG: race window entered (in-flight=1)' then 'Fatal trap 12 ... fault virtual address = 0x0', 'Stopped at crfree+0xc: cmpq $0,(%rdi)'", "findings/poc/DF-3120/run.log β patched kernel: 'SURVIVED ... EXITED_RC=0' over 300s (17.8M offs) + 120s confirmation, dmesg panic count 0", 'findings/poc/DF-3120/fix.diff β validated fix (git-apply-able, patch(1) -p1 verified byte-identical to tested kernel source)', 'findings/poc/DF-3120/env.txt, VERDICT.md, README.md']
PoC changes
Fixed QCMD packing (kernel Q_* constants are pre-shifted: cmds = Q_cmd<<8 | type, e.g. 0x10000 for user-quota Q_QUOTAON β the seed sketch's semantics double-shifted correctly by accident but sanity tools used 0x100 and got EINVAL); added 8 rotating quota files and a distinct /qtmnt/ctrlN control path per thread (a shared control file's namecache entry serializes the whole quotactl syscall and hides the race β with a shared path 1.5M quotactls produced zero window entries); unprivileged nobody children for Q_SYNC/Q_GETQUOTA/churn; children poll /tmp/QTSTOP (inherited flag never propagates to forked children); discovered stock kernel needs 'options QUOTA' (X86_64_QUOTA built in-guest via make nativekernel).
Verified recommended fix
Serialize ufs_quotaon/ufs_quotaoff with a per-mount lockmgr lock (um_quota_lock, lockinit in ffs_mountfs); wrap quotaoff body as ufs_quotaoff_locked so quotaon's internal swap-call doesn't self-deadlock; unlock the quota-file vp before taking the lock.
Verdict
Race double-execution of ufs_quotaoff proven on a QUOTA-enabled DragonFly 6.5-DEVELOPMENT x86_64 INVARIANTS kernel: two quotactl-capable threads passing the um_quotas[type]==NULLVP test at ufs_quota.c:503 both run the teardown. Clean baseline panicked <=45s with 'panic: VOP_STDCLOSE: BAD OPENCOUNT' (stack vn_close<-ufs_quotaoff<-ufs_quotactl) proving the double vn_close at :519; an instrumented run of the same race trapped at crfree+0xc with fault address 0x0, the crfree(NOCRED) NULL-deref at :521 (NOCRED==(struct ucred *)0). QTF_CLOSING (:505) is never tested by quotaoff itself and vfs_busy (ufs_vfsops.c:141) is a shared pin, so nothing serializes. Both manifestations match the source analysis. Privilege gate: every path in needs caps_priv_check(SYSCAP_NOQUOTA_WR) (or unmount-side quotaoff, which vfs_busy/MNTK_UNMOUNT serializes), so this is privileged-admin kernel corruption/DoS (Medium), not an unprivileged escalation; realistic ceiling is a delegated quota-capable process turning its narrow capability into a vnode UAF. Stock X86_64_GENERIC has no 'options QUOTA' (quotactl=EOPNOTSUPP), so a QUOTA kernel was built in-guest for verification. fix.diff (per-mount um_quota_lock; vp unlocked before lock in quotaon) was validated: identical PoC survived 300s/19M ops and a 120s confirmation with zero panics vs <=45s panic baseline.
No comments yet.