# DF-2786 — VERDICT

**Finding:** `mtx_abort_link()` stale-state double-unlink — `mtx_delete_link()`
leaves `MTX_LINK_LINKED_EX/SH` set on an already-unlinked link; the
`MTX_LINK_IDLE` reset happens later in `mtx_wait_link()` (kern_mutex.c:1023)
*outside* `MTX_LINKSPIN`, so a racing `mtx_abort_link()` can read the stale
LINKED state and execute the de-link writes through the victim's dangling
`next`/`prev`.

**Runtime verdict: NOT REPRODUCED on this guest** (honest negative; see §3).
**Finding stands on source-level state-machine proof** (§1) plus 116k+
measured abort-vs-exit window overlaps (§3). Severity Medium, confidence
likely.

---

## 1. The bug, by construction (path:line)

State machine of one `mtx_link` (only in-tree user: NFS, `rep->r_link`,
sys/vfs/nfs/nfs_socket.c:1311, aborted by `nfs_hardterm()` :2050, waited by
`nfs_rcvlock()` :2184):

```
IDLE --lock attempt--> LINKED_EX/SH --grant--> ACQUIRED --owner--> IDLE
                            |`-abort--> ABORTED --owner--> IDLE
                            `-timeout/EINTR--> mtx_delete_link() --> IDLE(:1023)
```

All list mutation is serialized by `MTX_LINKSPIN` — **except** the final
`link->state = MTX_LINK_IDLE` at `mtx_wait_link()` kern_mutex.c:1023, which
runs *after* `mtx_delete_link()` released LINKSPIN at :939/:940.

`mtx_delete_link()` (:914-941) unlinks the victim but **never writes
`link->state`** — the `MTX_LINK_LINKED_EX/SH` value survives the unlink.

`mtx_abort_link()` (:1046-1157) acquires LINKSPIN (:1056-1066), then switches
on `link->state`:

* `case MTX_LINK_LINKED_EX` (:1082): if `link->next != link`, executes
  `link->next->prev = link->prev; link->prev->next = link->next;`
  (:1095-1096) — **even when the victim was already unlinked** by a
  timeout/EINTR delete whose LINKSPIN window just closed. The writes go
  through the victim's *stale* neighbor pointers.
* `case MTX_LINK_LINKED_SH` (:1113): mirror image (:1126-1127).

Consequences of the stale write pair, in increasing severity:

1. **Idempotent (benign):** neighbors unchanged since the victim's unlink —
   the writes re-store current values.
2. **Write into freed/reused memory:** a former neighbor (P) was itself
   unlinked and — in the NFS usage — its `nfsreq` freed
   (nfs_socket.c:1457/:1219) and the chunk reallocated. The stale write
   `P->next = <other neighbor>` corrupts the new tenant; the live list keeps
   a pointer into freed memory; subsequent
   `mtx_chain_link_ex/sh` walks (grant/delete/addcount) trip
   `KKASSERT(link->state == MTX_LINK_LINKED_EX)` (:771/:847), `KKASSERT(link
   != NULL)` (:761), or dereference garbage → panic or silent corruption.
3. **Live-list poisoning:** neighbor P still queued but its successor
   changed; `P.next` is overwritten with the stale N, wiring the circular
   list into a wrong cycle → lost waiters (deadlock), double-grants (lock
   accounting corruption), or infinite walk in `mtx_chain_link_sh`'s
   mass-wakeup loop (:846-866) holding LINKSPIN (all-CPU wedge).

The interleaving is not excluded by any fence or lock: the aborter's
`atomic_cmpset_int` LINKSPIN acquisition and state read are only ordered
against LINKSPIN *holders*, and the deleting thread performs no store to
`link->state` between releasing LINKSPIN (:939) and :1023.

Reachability (in-tree): unprivileged local user on a system with a mounted
NFS filesystem (default `intr`-capable client paths): requester threads exit
`mtx_lock_ex_link` with EINTR/EWOULDBLOCK (PCATCH + `2*hz` timeouts,
nfs_socket.c:2184-2205) while `nfs_timer`/`nfs_nmcancelreqs` call
`nfs_softterm` → `nfs_hardterm` → `mtx_abort_link` on the same `rep->r_link`
— precisely the delete-exit window.

Why it is a *different* finding than DF-0047: DF-0047 is the *owner* racing
a **grant** during its delete window (leak of the granted lock). DF-2786 is
the **aborter** racing the owner's delete tail and re-executing de-link
writes through stale pointers (use-after-free write / list poisoning). Same
function neighborhood, different actor, different primitive, different fix.

## 2. PoC harness (mtx_abuse.c, v6 final)

KLD storm: contenders (fresh static double-buffered links per attempt,
`mtx_lock_ex_link` with short timeouts → constant `mtx_delete_link`
traffic), long-hold holders (deep queue, same-tick timeout batches),
aborters (`mtx_abort_link` on random live targets — nfs_hardterm-style).
Detection:

* retired links keep `next/prev = &mtxab_load` (module text, RX) and
  `state = 0x51dead51`; a scanner thread reports any kernel write into a
  retired link's `next`/`prev` (`CORRUPTION:` lines, `CORRUPT=` counter) —
  zero-false-positive detector for the stale-write primitive;
* INVARIANTS KKASSERTs / write-fault panics in kern_mutex.c would name the
  corrupted walk;
* `race_suspect` counts aborts that overlapped an owner's exit path
  (pre-abort `gen==0`, post-abort `gen!=0`).

Controls: `mtxabuse.noabort=1` runs identical traffic minus
`mtx_abort_link` — must stay `CORRUPT=0` and stable (it did, 90s + 60s).

## 3. What happened on the guest (full history, including my own mistakes)

Stock INVARIANTS kernel `6.5-DEVELOPMENT #0`, 6 vCPU KVM.

* **v1/v2/v4 runs (4 executions):** console ended ≤1s after aborters
  engaged; guest hard-wedged, no panic text. Forensics via QEMU monitor
  (my own instrumented QEMU, `info registers` per vCPU + offline symbol
  resolution against the guest kernel ELF): 5 CPUs in
  `_spin_lock_contested` on the *harness* `tlock` (value `0x400001` =
  4×SPINLOCK_EXCLWAIT|1), one CPU progressing. **Control run (noabort=1)
  reproduced the same wedge → the freezes were harness self-starvation**
  (6 busy-spinning kernel kthreads on 6 vCPUs, no yields), NOT kernel
  corruption. Reported here transparently; the harness was rewritten
  (fair yield/sleep discipline, then no harness locks at all).
* **v3 control (noabort, 90s):** 41,842 ops, 35,061 delete-path exits,
  4.66M aborter scans — stable, `CORRUPT=0`, clean SUMMARY, guest up.
* **v5 real (aborts on, 300s):** 16,062,889 aborts, 6,680,700 owner exits,
  115,662 measured abort-vs-exit window overlaps (~385/s) — **CORRUPT=0**,
  no panic, guest up. (Abort-hot churn keeps the queue shallow → the
  misfire branch is usually the harmless sole-link case.)
* **v6 real (batch-delete geometry, 300s):** 119,146 aborts, 134,726
  delete-path exits, 209 window overlaps — **CORRUPT=0**, stable.
* Also observed: `df47=0` in every run — DF-0047's grant-during-delete
  window never fired in these workloads either (both windows are narrow).

Aggregate abort-on exposure: ≈16.2M `mtx_abort_link` calls, ≈6.8M
delete-path exits, ≈116k measured window overlaps, 0 stale-write
manifestations.

**Why it doesn't land on this guest:** for the stale write to happen, the
aborter must complete a LINKSPIN `cmpset` *and* read `link->state` between
the owner's `atomic_clear_int` release (:939) and its `link->state` store
(:1023) — a ~3-6 instruction head start on the same cache line pair. On
real/KVM x86 the releasing CPU keeps the line and wins effectively always;
the misfire requires scheduler preemption of the owner inside that tail (or
an unusually slow release path). The overlapping aborts we *did* measure
(116k) arrived in the benign order (after :1023).

**Honest classification:** runtime `not_reproduced` / `impact none
(demonstrated)`. The race is real by construction (no ordering forbids the
interleaving; single-CPU preemption in the tail is all it takes — e.g. under
heavy interrupt load, different cache geometry, or future code changes in
that tail), hence the finding stands at **Medium / likely** with a 2-line
fix. On a busier machine (more CPUs, IRQ storms, preempting load) the window
is landable; we did not achieve it here within budget.

## 4. Fix

`fix.diff` (validated: applies cleanly to the guest's `/usr/src`; kernel
rebuilt and booted — see §5): terminate `link->state` to `MTX_LINK_IDLE`
inside `mtx_delete_link()`'s LINKED cases, *before* releasing LINKSPIN. A
racing abort then observes either LINKED (and de-links properly,
LINKSPIN-serialized) or IDLE (no de-link). The owner's later `:1023` store
becomes a redundant no-op. The DF-0047 default case (ACQUIRED observed) is
untouched.

## 5. Fix build/boot validation

* `patch -p1` on guest `/usr/src`: both hunks apply (kern_mutex.c:911/:931).
* `make nativekernel && make installkernel`: completed; guest rebooted into
  the patched kernel (uname in env.txt / run.log tail).
* Post-fix storm (same v6 harness, aborts on): stable, `CORRUPT=0`,
  no regressions. Since the baseline never manifested the corruption,
  this is a no-regression + build validation only — fix_status
  `inconclusive` (cannot diff against a reproducing baseline), the fix's
  correctness is by construction (§1/§4).