DF-2786 — mtx_abort_link stale-state double-unlink (kern_mutex.c)

Target: sys/kern/kern_mutex.c — mtx_delete_link() leaves link->state == MTX_LINK_LINKED_EX/SH set on an already-unlinked link; the reset to MTX_LINK_IDLE happens later in mtx_wait_link() (kern_mutex.c:1023) outside MTX_LINKSPIN. A concurrent mtx_abort_link() that acquires MTX_LINKSPIN in that window reads the stale LINKED state and executes the de-link writes (kern_mutex.c:1095-1096 / :1126-1127)

link->next->prev = link->prev;
link->prev->next = link->next;

through the victim's dangling next/prev — writing into whatever the stale neighbors are by then (in the only in-tree caller, NFS nfs_hardterm() → rep->r_link, the neighbors are other nfsreq r_link objects that may already be freed and reused), and poisoning the live circular wait queue.

Files

• mtx_abuse.c — KLD storm harness (final version: v6; see header)
• Makefile — kmod Makefile (build inside /usr/src/sys/modules/)
• build.sh — build commands (as executed on the guest)
• run.sh — run commands (as executed on the guest)
• build.log — full compiler output of the successful build
• run.log — decisive console output (baseline + control + real)
• env.txt — guest environment
• fix.diff — verified minimal fix (state termination under LINKSPIN)
• VERDICT.md — full narrative
• verdict.json — machine verdict
• manifest.json — artifact catalog

Build (on the DF guest, /usr/src present)

mkdir -p /usr/src/sys/modules/mtxabuse
cp mtx_abuse.c Makefile /usr/src/sys/modules/mtxabuse/
cd /usr/src/sys/modules/mtxabuse
env MAKESYSPATH=/usr/src/share/mk SYSDIR=/usr/src/sys make obj
env MAKESYSPATH=/usr/src/share/mk SYSDIR=/usr/src/sys make
cp /usr/obj/usr/src/sys/modules/mtxabuse/mtxabuse.ko /root/

Run

sysctl -w debug.debugger_on_panic=0
kenv mtxabuse.duration=300          # seconds; add mtxabuse.noabort=1 for control
kldload /root/mtxabuse.ko
# watch the serial console / dmesg for:
#   mtxabuse: PHASE1 ... NO-ABORT baseline    (must be clean)
#   mtxabuse: PHASE2 aborters engaged        (aborts start)
#   mtxabuse: CORRUPTION: kernel wrote retired link <p> next=... prev=...
#     ^ positive: kernel wrote into a retired (== freed in production) mtx_link
#   mtxabuse: SUMMARY ... CORRUPT=N
kldunload mtxabuse

Success criterion

CORRUPT > 0 (and/or mtxabuse: CORRUPTION: console lines, an INVARIANTS KKASSERT in mtx_chain_link_ex/sh, or a traced write-fault panic in mtx_abort_link), with the noabort=1 control run staying CORRUPT=0.

DF-2786 — VERDICT

Finding: mtx_abort_link() stale-state double-unlink — mtx_delete_link() leaves MTX_LINK_LINKED_EX/SH set on an already-unlinked link; the MTX_LINK_IDLE reset happens later in mtx_wait_link() (kern_mutex.c:1023) outside MTX_LINKSPIN, so a racing mtx_abort_link() can read the stale LINKED state and execute the de-link writes through the victim's dangling next/prev.

Runtime verdict: NOT REPRODUCED on this guest (honest negative; see §3). Finding stands on source-level state-machine proof (§1) plus 116k+ measured abort-vs-exit window overlaps (§3). Severity Medium, confidence likely.

1. The bug, by construction (path:line)

State machine of one mtx_link (only in-tree user: NFS, rep->r_link, sys/vfs/nfs/nfs_socket.c:1311, aborted by nfs_hardterm() :2050, waited by nfs_rcvlock() :2184):

IDLE --lock attempt--> LINKED_EX/SH --grant--> ACQUIRED --owner--> IDLE
                            |`-abort--> ABORTED --owner--> IDLE
                            `-timeout/EINTR--> mtx_delete_link() --> IDLE(:1023)

All list mutation is serialized by MTX_LINKSPIN — except the final link->state = MTX_LINK_IDLE at mtx_wait_link() kern_mutex.c:1023, which runs after mtx_delete_link() released LINKSPIN at :939/:940.

mtx_delete_link() (:914-941) unlinks the victim but never writes link->state — the MTX_LINK_LINKED_EX/SH value survives the unlink.

mtx_abort_link() (:1046-1157) acquires LINKSPIN (:1056-1066), then switches on link->state:

• case MTX_LINK_LINKED_EX (:1082): if link->next != link, executes link->next->prev = link->prev; link->prev->next = link->next; (:1095-1096) — even when the victim was already unlinked by a timeout/EINTR delete whose LINKSPIN window just closed. The writes go through the victim's stale neighbor pointers.
• case MTX_LINK_LINKED_SH (:1113): mirror image (:1126-1127).

Consequences of the stale write pair, in increasing severity:

1. Idempotent (benign): neighbors unchanged since the victim's unlink — the writes re-store current values.
2. Write into freed/reused memory: a former neighbor (P) was itself unlinked and — in the NFS usage — its nfsreq freed (nfs_socket.c:1457/:1219) and the chunk reallocated. The stale write P->next = <other neighbor> corrupts the new tenant; the live list keeps a pointer into freed memory; subsequent mtx_chain_link_ex/sh walks (grant/delete/addcount) trip KKASSERT(link->state == MTX_LINK_LINKED_EX) (:771/:847), KKASSERT(link != NULL) (:761), or dereference garbage → panic or silent corruption.
3. Live-list poisoning: neighbor P still queued but its successor changed; P.next is overwritten with the stale N, wiring the circular list into a wrong cycle → lost waiters (deadlock), double-grants (lock accounting corruption), or infinite walk in mtx_chain_link_sh's mass-wakeup loop (:846-866) holding LINKSPIN (all-CPU wedge).

The interleaving is not excluded by any fence or lock: the aborter's atomic_cmpset_int LINKSPIN acquisition and state read are only ordered against LINKSPIN holders, and the deleting thread performs no store to link->state between releasing LINKSPIN (:939) and :1023.

Reachability (in-tree): unprivileged local user on a system with a mounted NFS filesystem (default intr-capable client paths): requester threads exit mtx_lock_ex_link with EINTR/EWOULDBLOCK (PCATCH + 2*hz timeouts, nfs_socket.c:2184-2205) while nfs_timer/nfs_nmcancelreqs call nfs_softterm → nfs_hardterm → mtx_abort_link on the same rep->r_link — precisely the delete-exit window.

Why it is a different finding than DF-0047: DF-0047 is the owner racing a grant during its delete window (leak of the granted lock). DF-2786 is the aborter racing the owner's delete tail and re-executing de-link writes through stale pointers (use-after-free write / list poisoning). Same function neighborhood, different actor, different primitive, different fix.

2. PoC harness (mtx_abuse.c, v6 final)

KLD storm: contenders (fresh static double-buffered links per attempt, mtx_lock_ex_link with short timeouts → constant mtx_delete_link traffic), long-hold holders (deep queue, same-tick timeout batches), aborters (mtx_abort_link on random live targets — nfs_hardterm-style). Detection:

• retired links keep next/prev = &mtxab_load (module text, RX) and state = 0x51dead51; a scanner thread reports any kernel write into a retired link's next/prev (CORRUPTION: lines, CORRUPT= counter) — zero-false-positive detector for the stale-write primitive;
• INVARIANTS KKASSERTs / write-fault panics in kern_mutex.c would name the corrupted walk;
• race_suspect counts aborts that overlapped an owner's exit path (pre-abort gen==0, post-abort gen!=0).

Controls: mtxabuse.noabort=1 runs identical traffic minus mtx_abort_link — must stay CORRUPT=0 and stable (it did, 90s + 60s).

3. What happened on the guest (full history, including my own mistakes)

Stock INVARIANTS kernel 6.5-DEVELOPMENT #0, 6 vCPU KVM.

• v1/v2/v4 runs (4 executions): console ended ≤1s after aborters engaged; guest hard-wedged, no panic text. Forensics via QEMU monitor (my own instrumented QEMU, info registers per vCPU + offline symbol resolution against the guest kernel ELF): 5 CPUs in _spin_lock_contested on the harness tlock (value 0x400001 = 4×SPINLOCK_EXCLWAIT|1), one CPU progressing. Control run (noabort=1) reproduced the same wedge → the freezes were harness self-starvation (6 busy-spinning kernel kthreads on 6 vCPUs, no yields), NOT kernel corruption. Reported here transparently; the harness was rewritten (fair yield/sleep discipline, then no harness locks at all).
• v3 control (noabort, 90s): 41,842 ops, 35,061 delete-path exits, 4.66M aborter scans — stable, CORRUPT=0, clean SUMMARY, guest up.
• v5 real (aborts on, 300s): 16,062,889 aborts, 6,680,700 owner exits, 115,662 measured abort-vs-exit window overlaps (~385/s) — CORRUPT=0, no panic, guest up. (Abort-hot churn keeps the queue shallow → the misfire branch is usually the harmless sole-link case.)
• v6 real (batch-delete geometry, 300s): 119,146 aborts, 134,726 delete-path exits, 209 window overlaps — CORRUPT=0, stable.
• Also observed: df47=0 in every run — DF-0047's grant-during-delete window never fired in these workloads either (both windows are narrow).

Aggregate abort-on exposure: ≈16.2M mtx_abort_link calls, ≈6.8M delete-path exits, ≈116k measured window overlaps, 0 stale-write manifestations.

Why it doesn't land on this guest: for the stale write to happen, the aborter must complete a LINKSPIN cmpset and read link->state between the owner's atomic_clear_int release (:939) and its link->state store (:1023) — a ~3-6 instruction head start on the same cache line pair. On real/KVM x86 the releasing CPU keeps the line and wins effectively always; the misfire requires scheduler preemption of the owner inside that tail (or an unusually slow release path). The overlapping aborts we did measure (116k) arrived in the benign order (after :1023).

Honest classification: runtime not_reproduced / impact none (demonstrated). The race is real by construction (no ordering forbids the interleaving; single-CPU preemption in the tail is all it takes — e.g. under heavy interrupt load, different cache geometry, or future code changes in that tail), hence the finding stands at Medium / likely with a 2-line fix. On a busier machine (more CPUs, IRQ storms, preempting load) the window is landable; we did not achieve it here within budget.

4. Fix

fix.diff (validated: applies cleanly to the guest's /usr/src; kernel rebuilt and booted — see §5): terminate link->state to MTX_LINK_IDLE inside mtx_delete_link()'s LINKED cases, before releasing LINKSPIN. A racing abort then observes either LINKED (and de-links properly, LINKSPIN-serialized) or IDLE (no de-link). The owner's later :1023 store becomes a redundant no-op. The DF-0047 default case (ACQUIRED observed) is untouched.

5. Fix build/boot validation

• patch -p1 on guest /usr/src: both hunks apply (kern_mutex.c:911/:931).
• make nativekernel && make installkernel: completed; guest rebooted into the patched kernel (uname in env.txt / run.log tail).
• Post-fix storm (same v6 harness, aborts on): stable, CORRUPT=0, no regressions. Since the baseline never manifested the corruption, this is a no-regression + build validation only — fix_status inconclusive (cannot diff against a reproducing baseline), the fix's correctness is by construction (§1/§4).