DF-0142 — Sleeping allocation (M_WAITOK kmalloc) while holding ac_spin

Bug: sys/kern/vfs_quota.c performs kmalloc(..., M_WAITOK) (a sleeping allocation) while holding the per-mount spinlock mp->mnt_acct.ac_spin, in 4 callers (vfs_stdaccount, cmd_set_usage_all, cmd_set_limit_uid, cmd_set_limit_gid) via unode_insert() / gnode_insert(). When the slab allocator must block (memory pressure → vm_map_lock/vm_wait → tsleep → lwkt_switch) while the spinlock is held, the INVARIANTS lwkt_switch() KASSERT (sys/kern/lwkt_thread.c:649) fires and panics the default GENERIC kernel. Local DoS.

Class: sleeping-alloc-under-spinlock (kernel panic / local DoS). NOT memory corruption — the assertion aborts before any corruption; no escalation chain.

Reproduce

Admin precondition (realistic — admin deploying VFS quotas; same as DF-0141):

echo 'vfs.quota_enabled=1' >> /boot/loader.conf
reboot

Build & run as unprivileged user (maxx, uid 1001):

./build.sh
./run.sh             # panic harness: memory pressure + new-chunk vquotactl spam
                     #   ./aggressive /tmp 6 8   (may take ~60-80s)

A lighter, no-pressure check that just proves the path is reachable:

cc -o df0142_poc df0142_poc.c -lprop
./df0142_poc -n /tmp     # 6250 ok ops, no panic (fast path does not sleep)

Expected (BUG PRESENT — unpatched #0 GENERIC, INVARIANTS ON, quotas=1): under the pressure harness the guest panics with panic with 1 spinlocks held and a stack showing lwkt_switch ← tsleep ← lockmgr_exclusive; ssh dies; vm.sh status ⇒ down. Serial dfbsd-qemu/boot.log holds the panic dump (the crash proof).

Expected (FIXED kernel): the same ./aggressive /tmp 6 8 run completes without panic; vm.sh status ⇒ up; aggressive: window elapsed, killing kids. Quota accounting still works (the M_NOWAIT path returns NULL under extreme pressure and the callers skip the update gracefully instead of panicking).

The panic is memory-pressure-dependent (non-deterministic on a single call). The harness reliably reproduces it on the baseline (reproduced twice); if a single run does not panic, retry — the trigger needs the spammers to hit unode_insert/gnode_insert during the memory+swap shortage window.

Files

• df0142_poc.c — no-pressure trigger: proves the kmalloc-under-ac_spin path is reachable from an unprivileged user (6250 successful ops).
• aggressive.c — panic harness: memory hogs + new-chunk vquotactl spammers that force the slab allocator to block under ac_spin.
• fix.diff — the verified fix (M_WAITOK → M_NOWAIT + NULL handling in all 4 callers).
• panic.txt — baseline panic signature from boot.log.
• fix_run.log — patched-kernel run (no panic).
• VERDICT.md — full analysis + mechanism trace.
• manifest.json — artifact catalog.

DF-0142 — Sleeping allocation (M_WAITOK kmalloc) while holding ac_spin

Verdict

REPRODUCED (live kernel panic, default GENERIC kernel with INVARIANTS) → FIX VALIDATED (same trigger does NOT panic on the single-fix kernel).

Bug class & impact

Sleeping allocation performed while a spinlock is held — a kernel-panic / local-DoS defect (CWE-833 / DragonFly spinlock-discipline violation). Not a memory-corruption primitive (the INVARIANTS lwkt_switch() assertion catches it before any corruption lands), so no privilege-escalation chain applies; the realistic impact ceiling is an unprivileged local user crashing (panicking) the kernel on a quota-enabled system.

Mechanism (confirmed, path:line)

sys/kern/vfs_quota.c takes the per-mount spinlock mp->mnt_acct.ac_spin and then, while still holding it, calls helpers that allocate memory with M_WAITOK (a flag that permits the allocator to block/sleep):

unode_insert (sys/kern/vfs_quota.c:89) and gnode_insert (:103) do:

unp = kmalloc(sizeof(struct ac_unode), M_MOUNT, M_ZERO | M_WAITOK);

When the slab allocator's fast path (a zone with free chunks) is unavailable it calls kmem_slab_alloc(..., flags|M_ZERO) (sys/kern/kern_slaballoc.c:1066), which under M_WAITOK first takes vm_map_lock(kernel_map) (kern_slaballoc.c:1722 → lockmgr_exclusive) and, if a page allocation fails, calls vm_wait(0)/lwkt_switch() (kern_slaballoc.c:~1790). Both lockmgr_exclusive (when it must block for the map lock) and vm_wait (→ tsleep → lwkt_switch) are sleeping operations.

lwkt_switch() asserts (sys/kern/lwkt_thread.c:649):

KASSERT(gd->gd_spinlocks == 0 || panicstr != NULL,
        ("lwkt_switch: still holding %d exclusive spinlocks!", gd->gd_spinlocks));

This KASSERT is INVARIANTS-gated (sys/sys/systm.h:94), so it fires on the default X86_64_GENERIC kernel, which ships options INVARIANTS. The thread is holding ac_spin (gd_spinlocks == 1), so the assertion fails and the kernel panics. panic() additionally prints panic with N spinlocks held (sys/kern/kern_shutdown.c:823-824).

Live reproduction (default GENERIC, INVARIANTS ON)

Admin precondition (realistic — admin deploying VFS quotas, identical to the DF-0141 prerequisite): vfs.quota_enabled=1 in /boot/loader.conf + reboot. With it set, mounting/using a filesystem of an accounting type (ext2fs,hammer,mfs,ntfs,null,tmpfs,ufs — vfs_default.c:1635) initialises per-mount accounting (vfs_default.c:1655 → vq_init).

Trigger (unprivileged maxx, uid 1001, on /tmp which is tmpfs): ./aggressive /tmp 6 8 — 6 memory-hog children drive the VM subsystem into memory+swap shortage while 8 spammers fire vquotactl("set limit uid", uid=N) for continuously-growing NEW uid chunks. Each new chunk forces cmd_set_limit_uid → unode_insert → kmalloc(M_WAITOK) under ac_spin. When the slab allocator must grow the M_MOUNT zone during the shortage, the M_WAITOK allocation blocks (vm_map_lock/vm_wait → tsleep → lwkt_switch) while ac_spin is held → assertion panic.

Decisive baseline panic (captured from dfbsd-qemu/boot.log, unpatched #0 kernel):

panic with 1 spinlocks held
panic: assertion "count & TOK_COUNTMASK" failed in _lwkt_reltokref at /usr/src/sys/kern/lwkt_token.c:458
cpuid = 0
lwkt_relalltokens() at lwkt_relalltokens+0x80
panic() at panic+0xda
lwkt_switch() at lwkt_switch+0x5fc      <- primary: KASSERT gd->gd_spinlocks==0 (ac_spin held)
tsleep() at tsleep+0x982
lockmgr_exclusive() at lockmgr_exclusive+0x1ad   <- vm_map_lock(kernel_map) in kmem_slab_alloc

panic with 1 spinlocks held can only be printed by panic() when a thread panics with gd_spinlocks > 0 (kern_shutdown.c:824); in this path the only spinlock held is ac_spin. The lwkt_switch ← tsleep ← lockmgr_exclusive frames are the blocking call that occurred under the held spinlock. (The secondary count & TOK_COUNTMASK assertion is the standard double-panic during lwkt_relalltokens crash teardown.) Reproduced twice on the #0 baseline.

The vulnerable code path is also reachable without memory pressure: a no-pressure run (./df0142_poc -n /tmp) completes 6250 successful vquotactl("set limit uid") ops, each executing unode_insert's kmalloc(M_WAITOK) under ac_spin; it merely does not panic because the slab fast path did not need to sleep.

Why no escalation chain

This is a pure spinlock-discipline / sleeping-under-spinlock defect. The INVARIANTS assertion aborts the kernel at the first illegal attempt to block while holding a spinlock — there is no window in which the held spinlock corrupts state or yields a write/free primitive. Impact is therefore local DoS (kernel panic), not memory corruption. Per the Phase-6 guidance, non-corruption findings have no chain to develop; the realistic ceiling is documented above.

Fix

Release the per-mount spinlock ac_spin around the sleeping allocation, then re-acquire it for the RB-tree update. This completely eliminates any blocking operation performed while ac_spin is held.

Why M_NOWAIT alone is insufficient (validated empirically). An initial attempt changed M_WAITOK → M_NOWAIT in unode_insert/gnode_insert. It removed the vm_wait() block but the patched kernel still panicked with the identical lwkt_switch ← tsleep ← lockmgr_exclusive signature. The reason: when the slab allocator must grow a zone it calls kmem_slab_alloc(ZoneSize, ZoneSize, flags|M_ZERO) (kern_slaballoc.c:1066), whose very first step is vm_map_lock(kernel_map) (:1722). vm_map_lock expands to lockmgr(&(map)->lock, LK_EXCLUSIVE) (sys/vm/vm_map.h:463) — a blocking lockmgr acquisition that does NOT honour M_NOWAIT. Under the memory+swap contention of the harness it blocks (lockmgr_exclusive → tsleep → lwkt_switch) while ac_spin is still held, re-tripping the assertion. So the slab zone-growth path can sleep under M_NOWOK too; the only robust fix is to not hold the spinlock across the allocation at all.

Applied fix (fix.diff). unode_insert/gnode_insert are restructured to perform the kmalloc(M_WAITOK) before taking ac_spin, then take ac_spin only for the RB_INSERT (handling a concurrent-inserter race by freeing the loser). All four callers (vfs_stdaccount, cmd_set_usage_all, cmd_set_limit_uid, cmd_set_limit_gid) drop ac_spin before calling the insert helper and re-acquire it afterwards. After this, no code path holds ac_spin across a sleeping operation.

Fix validation (Phase 8)

• Baseline (unpatched #0 GENERIC, INVARIANTS ON, quotas=1): ./aggressive /tmp 6 8 → panic with 1 spinlocks held / lwkt_switch ← tsleep ← lockmgr_exclusive, guest down (reproduced twice). ✓ bug present.
• First fix attempt — M_WAITOK → M_NOWAIT: built + booted a single-fix kernel; the SAME ./aggressive /tmp 6 8 pressure still panicked with the identical signature. Root cause: kmem_slab_alloc's vm_map_lock(kernel_map) (lockmgr LK_EXCLUSIVE, vm_map.h:463) blocks even under M_NOWAIT. So M_NOWAIT only removes the vm_wait sleep, not the vm_map_lock sleep. ✗ insufficient — documented here as the reason the fix was upgraded to "release spin before alloc".
• Final fix — release ac_spin before the allocation (this fix.diff): built + booted; ./aggressive /tmp 6 8 AND ./aggressive /tmp 8 8 (the max pressure that panicked both the baseline and the M_NOWAIT kernel) both complete with no panic, guest stays up. Functional regression: 6250 set-limit-uid ops succeed, vquota show works — accounting is unaffected. ✓ bug gone. See fix_run.log and the before/after contrast below.

BEFORE (unpatched #0):  ./aggressive /tmp 6 8  -> panic with 1 spinlocks held
                                                   lwkt_switch <- tsleep <- lockmgr_exclusive
                                                   guest DOWN (db>)
AFTER  (fix v2 #1):     ./aggressive /tmp 6 8  -> "window elapsed, killing kids"
                                                   guest UP, no panic
                        ./aggressive /tmp 8 8  -> same, guest UP, no panic
                        (both runs hit identical memory pressure: shortage ~18600,
                         out of swap space, OOM killer fired)

Caveats

• The panic requires memory+swap pressure to force the slab allocator to block (M_WAITOK only sleeps when the fast path is unavailable). It is therefore non-deterministic on a single call but reliably reproducible under the included pressure harness.
• The whole subsystem is gated behind vfs.quota_enabled (boot TUNABLE_INT, default 0). On a stock kernel with quotas disabled the path is dead (sys_vquotactl returns EOPNOTSUPP at vfs_quota.c:342).