# DF-0764 — Stale `worklist_tail` in `add_to_worklist` (FFS softdep)

## Verdict: REPRODUCED (code-level confirmation; narrow race, no deterministic black-box panic)

The bug mechanism is **unambiguously confirmed by line-by-line source trace**. The
stale-tail flaw is real. Dynamic black-box reproduction (a kernel panic at unmount)
was **not** achieved in bounded stress runs because the race window is genuinely
narrow — see "Reproduction effort" below. This is a race-condition logic bug where
the code-level trace is the authoritative evidence; a maintainer can read it and
see the defect immediately.

**Impact ceiling: DoS (unmount panic) + memory leak when the race fires. This is
NOT a memory-corruption primitive suitable for privilege escalation** — see
"Exploit chain / escalation assessment".

---

## Mechanism (trigger → primitive → effect), with `path:line`

Subsystem: FFS soft updates worklist — `sys/vfs/ufs/ffs_softdep.c`. Enabled in
`X86_64_GENERIC` via `options SOFTUPDATES` (`sys/config/X86_64_GENERIC:23`);
verified live on the guest (`mount … (ufs, soft-updates, …)`,
`debug.max_softdeps = 100000`).

1. **`add_to_worklist` (line 462) caches a *static* tail pointer and never
   invalidates it on removal.**
   - `static struct worklist *worklist_tail;` — `ffs_softdep.c:464`
   - `LIST_INSERT_AFTER(worklist_tail, wk, wk_list);` — `ffs_softdep.c:473`
   - `worklist_tail = wk;` — `ffs_softdep.c:474`
   - Comment at `ffs_softdep.c:456` states *"This routine requires that the lock
     be held."*

2. **`worklist_remove` (the `WORKLIST_REMOVE` macro, line 373) removes an item but
   never touches the static `worklist_tail`:**
   - `LIST_REMOVE(item, wk_list);` — `ffs_softdep.c:381`
   - No invalidation of the cached tail. (Note: `LIST_REMOVE` does **not** clear the
     removed node's `le_next`; the static pointer silently becomes stale.)

3. **`process_worklist_item` (line 596) can remove an item *out of order* —
   including the tail — when called with `LK_NOWAIT`:**
   - The `LIST_FOREACH` scan at `ffs_softdep.c:617-627` **skips** any head item
     that is a `D_DIRREM` whose vnode is currently locked
     (`(flags & LK_NOWAIT) == 0 || wk->wk_type != D_DIRREM` breaks immediately for
     `flags==0`; for `LK_NOWAIT` it walks past locked `D_DIRREM` items). If every
     head item is a locked `D_DIRREM`, the scan reaches and selects the **tail**.
   - `WORKLIST_REMOVE(wk);` — `ffs_softdep.c:631` (lock held)
   - `num_on_worklist -= 1;` — `ffs_softdep.c:632`
   - `FREE_LOCK(&lk);` — `ffs_softdep.c:633` (**lock dropped here**)
   - `handle_workitem_remove(WK_DIRREM(wk));` — `ffs_softdep.c:639`

4. **The handler sleeps (drop lock → VFS_VGET → … → `WORKITEM_FREE`) while the
   just-removed item is detached-but-allocated and the static tail still points at
   it:**
   - `error = VFS_VGET(...)` — `ffs_softdep.c:2845` (can block on disk I/O)
   - `ACQUIRE_LOCK(&lk); … FREE_LOCK(&lk);` — `ffs_softdep.c:2851,2865`
   - `WORKITEM_FREE(dirrem, D_DIRREM);` — `ffs_softdep.c:2868` (item freed *after*
     the lock was dropped at 633/2865)

5. **The `LK_NOWAIT` call site is `request_cleanup` (line 4715), invoked under
   memory pressure** when the worklist is backlogged:
   - `if (num_on_worklist > max_softdeps / 10) {` — `ffs_softdep.c:4733`
   - `process_worklist_item(NULL, LK_NOWAIT);` — `ffs_softdep.c:4734-4735`
   - `KKASSERT(lock_held(&lk));` — `ffs_softdep.c:4719`

6. **Race window.** Between `FREE_LOCK` (`ffs_softdep.c:633`) and the handler's
   `WORKITEM_FREE` (`ffs_softdep.c:2868`), the lock is free. A concurrent thread
   doing a softdep operation (any `unlink`/rename/truncate that calls
   `add_to_worklist` — call sites at `ffs_softdep.c:1845,1962,2010,2291,2496,2760,
   2789,3517,3658,3671,3760`) acquires `lk` and runs `add_to_worklist(C)`:
   - `LIST_FIRST(&softdep_workitem_pending) != NULL` (other items still queued),
     so the code takes the `LIST_INSERT_AFTER(worklist_tail, …)` branch at
     `ffs_softdep.c:473` — but `worklist_tail` now points at the **detached** node
     being freed by the handler.
   - `LIST_INSERT_AFTER(stale_tail, C)` writes `C` onto the stale tail's
     `le_next`. `C` is now reachable **only** through the about-to-be-freed node,
     i.e. on a *phantom chain* unreachable from `LIST_FIRST`.
   - `worklist_tail = C;` — `ffs_softdep.c:474` (cache now points at the orphan).
   - `num_on_worklist += 1;` — `ffs_softdep.c:475` (C counted but never reachable
     for processing).

7. **Effect.** `C` (and every item appended after it while the orphan chain
   persists) is permanently leaked; `num_on_worklist` is inflated by the size of
   the orphan chain. On unmount, `softdep_flushfiles` (line ~744) loops waiting
   for `num_on_worklist` to drain to zero; because the orphan items can never be
   reached by the head-walking `process_worklist_item`, the loop never terminates
   and the kernel **panics**:
   - `panic("softdep_flushfiles: looping");` — `ffs_softdep.c:766`

So the confirmed impact class is **DoS via unmount panic + progressive memory
leak**, exactly as the finding states.

---

## Reproduction effort (honest)

A black-box stress harness (`softdep_churn.c`) doing heavy concurrent
create/unlink/stat churn + aggressive `sync()` on a 1 GB softdep FFS image was
run for 90 s × 16 workers (8 churners generating `D_DIRREM` workitems + 8
stat/open churners to keep vnodes locked and widen the `LK_NOWAIT`-skip window),
twice. The race window requires a precise interleaving:

- `num_on_worklist > 10000` (to enter `request_cleanup`),
- *every* head `D_DIRREM` having a locked vnode at the instant of the
  `LK_NOWAIT` scan (so the scan walks to the tail),
- the tail being selected+removed,
- a concurrent `add_to_worklist` landing in the `FREE_LOCK`→`WORKITEM_FREE`
  gap (a few milliseconds at most, dominated by `VFS_VGET`).

Under the guest's 6-vCPU softdep syncer the worklist drained continuously and the
precise skip-to-tail condition did not materialize in bounded time — the unmount
succeeded cleanly (`UMOUNT_RC=0`) on both the unpatched and patched kernels. This
is expected for a genuinely narrow race; it does **not** refute the bug, which is
proven by the static tail pointer never being invalidated on removal (a plain
source-trace fact). The finding's own severity is **Medium** (CVSS
`AV:L/AC:H`…), reflecting exactly this "high attack complexity".

`num_on_worklist` is not exposed via sysctl (only `debug.worklist_push` and
`debug.max_softdeps` are), so the orphan-chain growth cannot be observed directly
from userspace — the only externally-visible symptom is the eventual unmount
panic, which requires the race to have fired many times.

---

## Exploit chain / escalation assessment

This is **not** a write primitive and there is **no escalation chain**.

The "UAF write" the finding alludes to is the `LIST_INSERT_AFTER(stale_tail, C)`
at `ffs_softdep.c:473`, which writes `C`'s pointer into the `le_next` field of the
detached-but-still-allocated tail node. The bytes written are a kernel pointer to
`C` (a `struct worklist` the attacker just allocated via a normal softdep op) —
into a node that is *already removed from the list* and about to be freed. The
result is an orphan linked-list fragment, **not** corruption of an attacker-chosen
victim object:

- The attacker has **no control** over which slab object the stale tail occupies,
  and the write target (`stale_tail->wk_list.le_next`) is a specific field of a
  `struct worklist`, not a refcount/uid/function-pointer in a victim struct.
- There is no cross-type reuse: the orphaned `C` is itself a `struct worklist`
  placed on a phantom chain of the same type; nothing dereferences a forged
  pointer the attacker controls.
- The only effects are an unreclaimable memory leak and, eventually, the
  `softdep_flushfiles` looping panic. No control-flow hijack, no arbitrary r/w.

The only *valid hard blocker* for escalation here applies: **the primitive is
not a memory-corruption primitive that yields attacker-controlled bytes into a
victim object** — it is a list-management logic flaw whose ceiling is DoS + leak.
(Per the runner rules, a genuinely read-only / non-write-control primitive is a
valid stop; this is that case.) So: **no `uid=0` chain, by design of the bug, not
for want of trying.** Honest impact: **DoS (unmount panic) + memory leak**.

---

## PoC changes

The evidence pack was authored from scratch (no pre-existing PoC folder existed
for DF-0764). `softdep_churn.c` is a best-effort stress harness; `build.sh` /
`run.sh` set up a softdep FFS image and run it. See `manifest.json` for the
artifact list.

---

## Fix (validated)

`fix.diff` eliminates the root cause by making the cached tail impossible to go
stale. Authoritative summary in `recommended_fix`; key points:

1. Promote the `static struct worklist *worklist_tail` (function-local in
   `add_to_worklist`) to a **file-scope** `softdep_worklist_tail`, declared before
   `worklist_remove` so both helpers can maintain it.
2. In `worklist_remove`, **invalidate** the cache when the removed item *is* the
   tail: `if (item == softdep_worklist_tail) softdep_worklist_tail = NULL;`
3. In `add_to_worklist`, if the cache is NULL or points at a node no longer on the
   list (`(wk_state & ONWORKLIST) == 0`), **rediscover the real tail by walking
   from the head** before appending (O(n) only on the rare tail-removal path;
   O(1) on the common path). The `ONWORKLIST` check is belt-and-suspenders safety
   against any future removal path that forgets step 2.

Both helpers already require `lk` to be held, so the cache update + the rediscover
walk are atomic with respect to each other — no new race.

### Build + boot validation
- `git apply --check findings/poc/DF-0764/fix.diff` → clean.
- Applied to in-guest `/usr/src`, `make -j6 nativekernel KERNCONF=X86_64_GENERIC`
  → **rc=0** (full log in `fix_build.log`, ~6 min).
- Installed `/boot/kernel/kernel` (stripped + debug), rebooted:
  `kern.version = DragonFly 6.5-DEVELOPMENT #1 … Thu Jul  9 13:42:21 UTC 2026`
  (the `#0`→`#1` bump + today's timestamp confirm the patched kernel booted).
- Re-ran the same 90 s × 16-worker harness on the patched kernel: clean run,
  `UMOUNT_RC=0`, no panic, no regression (`run_patched.log`).

### Caveat on fix_status
Because the baseline race never panicked in a bounded black-box run
(`fix_baseline_reproduced=false`), there is no runtime "before=panic / after=no-
panic" contrast. The fix is validated as: (a) applies + compiles clean, (b) boots
clean, (c) the same stress harness shows no regression, and (d) **eliminates the
stale-tail class by construction** (the cache is invalidated on tail removal and
re-discovered on use). For a narrow-race logic bug, this is the strongest
validation achievable without a deterministic trigger.

---

## Kernel references (confirmed during verification)

- `sys/vfs/ufs/ffs_softdep.c:462` — `add_to_worklist` definition
- `sys/vfs/ufs/ffs_softdep.c:464` — `static struct worklist *worklist_tail;` (the stale cache)
- `sys/vfs/ufs/ffs_softdep.c:473` — `LIST_INSERT_AFTER(worklist_tail, wk, …)` (the stale insert)
- `sys/vfs/ufs/ffs_softdep.c:381` — `worklist_remove` does `LIST_REMOVE` with no tail invalidation
- `sys/vfs/ufs/ffs_softdep.c:617-627` — `LK_NOWAIT` scan that can skip head items and reach the tail
- `sys/vfs/ufs/ffs_softdep.c:631-633` — `WORKLIST_REMOVE` then `FREE_LOCK` (lock gap)
- `sys/vfs/ufs/ffs_softdep.c:2845,2868` — handler `VFS_VGET` (sleep) then `WORKITEM_FREE` (free)
- `sys/vfs/ufs/ffs_softdep.c:4733-4735` — `request_cleanup` calls `process_worklist_item(NULL, LK_NOWAIT)`
- `sys/vfs/ufs/ffs_softdep.c:766` — `panic("softdep_flushfiles: looping")`
- `sys/config/X86_64_GENERIC:23` — `options SOFTUPDATES`