# DF-0142 — Sleeping allocation (M_WAITOK kmalloc) while holding `ac_spin`

## Verdict
**REPRODUCED** (live kernel panic, default GENERIC kernel with INVARIANTS) → **FIX VALIDATED**
(same trigger does NOT panic on the single-fix kernel).

## Bug class & impact
**Sleeping allocation performed while a spinlock is held** — a kernel-panic /
local-DoS defect (CWE-833 / DragonFly spinlock-discipline violation). Not a
memory-corruption primitive (the INVARIANTS `lwkt_switch()` assertion catches
it before any corruption lands), so no privilege-escalation chain applies; the
realistic impact ceiling is **an unprivileged local user crashing (panicking)
the kernel** on a quota-enabled system.

## Mechanism (confirmed, path:line)

`sys/kern/vfs_quota.c` takes the per-mount spinlock `mp->mnt_acct.ac_spin` and
then, while still holding it, calls helpers that allocate memory with
`M_WAITOK` (a flag that permits the allocator to **block/sleep**):

| Caller (vfs_quota.c)        | spin_lock     | sleeping call site |
|-----------------------------|---------------|--------------------|
| `vfs_stdaccount`            | `:158`        | `unode_insert` `:163` / `gnode_insert` `:165` |
| `cmd_set_usage_all`         | `:228`        | `:254` / `:260`    |
| `cmd_set_limit_uid`         | `:298`        | `unode_insert` `:300` |
| `cmd_set_limit_gid`         | `:319`        | `gnode_insert` `:321` |

`unode_insert` (`sys/kern/vfs_quota.c:89`) and `gnode_insert` (`:103`) do:

```c
unp = kmalloc(sizeof(struct ac_unode), M_MOUNT, M_ZERO | M_WAITOK);
```

When the slab allocator's fast path (a zone with free chunks) is unavailable it
calls `kmem_slab_alloc(..., flags|M_ZERO)` (`sys/kern/kern_slaballoc.c:1066`),
which under `M_WAITOK` first takes `vm_map_lock(kernel_map)`
(`kern_slaballoc.c:1722` → `lockmgr_exclusive`) and, if a page allocation
fails, calls `vm_wait(0)`/`lwkt_switch()` (`kern_slaballoc.c:~1790`). Both
`lockmgr_exclusive` (when it must block for the map lock) and `vm_wait`
(`→ tsleep → lwkt_switch`) are sleeping operations.

`lwkt_switch()` asserts (`sys/kern/lwkt_thread.c:649`):

```c
KASSERT(gd->gd_spinlocks == 0 || panicstr != NULL,
        ("lwkt_switch: still holding %d exclusive spinlocks!", gd->gd_spinlocks));
```

This `KASSERT` is INVARIANTS-gated (`sys/sys/systm.h:94`), so it **fires on the
default `X86_64_GENERIC` kernel**, which ships `options INVARIANTS`. The thread
is holding `ac_spin` (`gd_spinlocks == 1`), so the assertion fails and the
kernel panics. `panic()` additionally prints `panic with N spinlocks held`
(`sys/kern/kern_shutdown.c:823-824`).

## Live reproduction (default GENERIC, INVARIANTS ON)

Admin precondition (realistic — admin deploying VFS quotas, identical to the
DF-0141 prerequisite): `vfs.quota_enabled=1` in `/boot/loader.conf` + reboot.
With it set, mounting/using a filesystem of an accounting type
(`ext2fs,hammer,mfs,ntfs,null,tmpfs,ufs` — `vfs_default.c:1635`) initialises
per-mount accounting (`vfs_default.c:1655 → vq_init`).

Trigger (unprivileged `maxx`, uid 1001, on `/tmp` which is `tmpfs`):
`./aggressive /tmp 6 8` — 6 memory-hog children drive the VM subsystem into
memory+swap shortage while 8 spammers fire `vquotactl("set limit uid", uid=N)`
for continuously-growing NEW uid chunks. Each new chunk forces
`cmd_set_limit_uid → unode_insert → kmalloc(M_WAITOK)` under `ac_spin`. When
the slab allocator must grow the `M_MOUNT` zone during the shortage, the
`M_WAITOK` allocation blocks (`vm_map_lock`/`vm_wait` → `tsleep` →
`lwkt_switch`) while `ac_spin` is held → assertion panic.

Decisive baseline panic (captured from `dfbsd-qemu/boot.log`, unpatched #0
kernel):

```
panic with 1 spinlocks held
panic: assertion "count & TOK_COUNTMASK" failed in _lwkt_reltokref at /usr/src/sys/kern/lwkt_token.c:458
cpuid = 0
lwkt_relalltokens() at lwkt_relalltokens+0x80
panic() at panic+0xda
lwkt_switch() at lwkt_switch+0x5fc      <- primary: KASSERT gd->gd_spinlocks==0 (ac_spin held)
tsleep() at tsleep+0x982
lockmgr_exclusive() at lockmgr_exclusive+0x1ad   <- vm_map_lock(kernel_map) in kmem_slab_alloc
```

`panic with 1 spinlocks held` can **only** be printed by `panic()` when a
thread panics with `gd_spinlocks > 0` (`kern_shutdown.c:824`); in this path the
only spinlock held is `ac_spin`. The `lwkt_switch ← tsleep ← lockmgr_exclusive`
frames are the blocking call that occurred under the held spinlock. (The
secondary `count & TOK_COUNTMASK` assertion is the standard double-panic during
`lwkt_relalltokens` crash teardown.) Reproduced twice on the #0 baseline.

The vulnerable code path is also **reachable without memory pressure**: a
no-pressure run (`./df0142_poc -n /tmp`) completes 6250 successful
`vquotactl("set limit uid")` ops, each executing `unode_insert`'s
`kmalloc(M_WAITOK)` under `ac_spin`; it merely does not panic because the slab
fast path did not need to sleep.

## Why no escalation chain
This is a pure spinlock-discipline / sleeping-under-spinlock defect. The
INVARIANTS assertion aborts the kernel at the first illegal attempt to block
while holding a spinlock — there is no window in which the held spinlock
corrupts state or yields a write/free primitive. Impact is therefore local DoS
(kernel panic), not memory corruption. Per the Phase-6 guidance, non-corruption
findings have no chain to develop; the realistic ceiling is documented above.

## Fix
Release the per-mount spinlock `ac_spin` around the sleeping allocation, then
re-acquire it for the RB-tree update. This completely eliminates any blocking
operation performed while `ac_spin` is held.

**Why `M_NOWAIT` alone is insufficient (validated empirically).** An initial
attempt changed `M_WAITOK → M_NOWAIT` in `unode_insert`/`gnode_insert`. It
removed the `vm_wait()` block but the patched kernel **still panicked** with
the identical `lwkt_switch ← tsleep ← lockmgr_exclusive` signature. The reason:
when the slab allocator must grow a zone it calls
`kmem_slab_alloc(ZoneSize, ZoneSize, flags|M_ZERO)` (`kern_slaballoc.c:1066`),
whose very first step is `vm_map_lock(kernel_map)` (`:1722`). `vm_map_lock`
expands to `lockmgr(&(map)->lock, LK_EXCLUSIVE)` (`sys/vm/vm_map.h:463`) — a
**blocking lockmgr acquisition that does NOT honour `M_NOWAIT`**. Under the
memory+swap contention of the harness it blocks (`lockmgr_exclusive → tsleep →
lwkt_switch`) while `ac_spin` is still held, re-tripping the assertion. So the
slab zone-growth path can sleep under `M_NOWOK` too; the only robust fix is to
not hold the spinlock across the allocation at all.

**Applied fix (`fix.diff`).** `unode_insert`/`gnode_insert` are restructured to
perform the `kmalloc(M_WAITOK)` **before** taking `ac_spin`, then take
`ac_spin` only for the `RB_INSERT` (handling a concurrent-inserter race by
freeing the loser). All four callers (`vfs_stdaccount`,
`cmd_set_usage_all`, `cmd_set_limit_uid`, `cmd_set_limit_gid`) drop `ac_spin`
before calling the insert helper and re-acquire it afterwards. After this,
no code path holds `ac_spin` across a sleeping operation.

## Fix validation (Phase 8)
- **Baseline (unpatched #0 GENERIC, INVARIANTS ON, quotas=1):** `./aggressive
  /tmp 6 8` → `panic with 1 spinlocks held` / `lwkt_switch ← tsleep ←
  lockmgr_exclusive`, guest down (reproduced twice). ✓ bug present.
- **First fix attempt — `M_WAITOK → M_NOWAIT`:** built + booted a single-fix
  kernel; the SAME `./aggressive /tmp 6 8` pressure **still panicked** with the
  identical signature. Root cause: `kmem_slab_alloc`'s `vm_map_lock(kernel_map)`
  (`lockmgr LK_EXCLUSIVE`, vm_map.h:463) blocks even under `M_NOWAIT`. So
  `M_NOWAIT` only removes the `vm_wait` sleep, not the `vm_map_lock` sleep.
  ✗ insufficient — documented here as the reason the fix was upgraded to
  "release spin before alloc".
- **Final fix — release `ac_spin` before the allocation (this `fix.diff`):**
  built + booted; `./aggressive /tmp 6 8` AND `./aggressive /tmp 8 8` (the max
  pressure that panicked both the baseline and the `M_NOWAIT` kernel) both
  **complete with no panic, guest stays up**. Functional regression: 6250
  `set-limit-uid` ops succeed, `vquota show` works — accounting is unaffected.
  ✓ bug gone. See `fix_run.log` and the before/after contrast below.

```
BEFORE (unpatched #0):  ./aggressive /tmp 6 8  -> panic with 1 spinlocks held
                                                   lwkt_switch <- tsleep <- lockmgr_exclusive
                                                   guest DOWN (db>)
AFTER  (fix v2 #1):     ./aggressive /tmp 6 8  -> "window elapsed, killing kids"
                                                   guest UP, no panic
                        ./aggressive /tmp 8 8  -> same, guest UP, no panic
                        (both runs hit identical memory pressure: shortage ~18600,
                         out of swap space, OOM killer fired)
```

## Caveats
- The panic requires memory+swap pressure to force the slab allocator to block
  (`M_WAITOK` only sleeps when the fast path is unavailable). It is therefore
  non-deterministic on a single call but reliably reproducible under the
  included pressure harness.
- The whole subsystem is gated behind `vfs.quota_enabled` (boot TUNABLE_INT,
  default 0). On a stock kernel with quotas disabled the path is dead
  (`sys_vquotactl` returns `EOPNOTSUPP` at `vfs_quota.c:342`).