DF-3061 — NFS rename retry corrupts/leaks namecache state → remote kernel panic

What

nfs_namei() (sys/vfs/nfs/nfs_subs.c:1003-1011) supports being called a second time on the same struct nlookupdata ("retry case"), but only rescues the path buffer. The subsequent nlookup_init_raw() (called at nfs_subs.c:1155) unconditionally bzero()s the whole nd, silently discarding live state: the exclusively-locked leaf ncp (nl_nch, NLC_NCPISLOCKED), its cache reference, the nl_rootnch/nl_jailnch references and a crhold()'d nl_cred (see nlookup_init_raw, sys/kern/vfs_nlookup.c:255-292, and nlookup_done, :366-400 for the release protocol that is being skipped).

The only in-tree caller that uses the retry is nfsrv_rename() (sys/vfs/nfs/nfs_serv.c:2255-2271): when the namecache topology changes between the two nfs_namei() lookups and cache_lock4_tondlocked() revalidation, it does cache_put(&fnchd); cache_put(&tnchd); goto again;. The retry branch additionally leaves fnchd_status/tnchd_status == 2 and stale fdirp/tdirp pointers, so a failure of the second iteration's lookups drives the nfsmout: cleanup (nfs_serv.c:2382-2391) to cache_unlock/cache_drop already-released handles and vrele an already-vrele'd directory vnode — double-drop / refcount-underflow / use-after-free.

Impact

• Observed on the stock INVARIANTS guest: kernel panic in the nfsd process immediately after the nfs - retry rename console message — Fatal trap 12 ... lockmgr_release+0x11, fault at 0xc8 (a NULL ncp's nc_lock at offset 0xc0 in struct namecache). 2/2 runs, ~2-2.5 min of racing each.
• Even without the crash: leaked exclusive ncp locks make every later lookup of the affected names block forever; leaked cache/cred refs are unbounded.
• Reachable by any NFS client permitted by the server's exports, with ordinary RENAME RPCs racing concurrent modifications of the same names (a local racer on the exported tree or a second client). No authentication beyond the export ACL is required.

Files

• racer.c — server-side churn racer (rename/unlink loop on src/dst)
• renamer.c, cl.c (built in-guest) — client-side RENAME hammer
• panic.txt, panic_run2.txt — the two captured panics
• fix.diff — the verified fix (nfs_subs.c retry teardown + nfs_serv.c status/pointer reset)
• build.log, run.log — fix-validation kernel build and race run

Reproduce

In the guest (NFS server + loopback client):

mkdir -p /export /mnt/nfs
echo "/export -maproot=root localhost" > /etc/exports
rpcbind; /sbin/nfsd -t -u -n 4; sleep 1; /sbin/mountd & sleep 2
mount localhost:/export /mnt/nfs
# build racer + cl (see run.sh), then:
( /tmp/rc4 /export & ) ; ( for i in 1 2 3 4 5 6; do /tmp/cl /mnt/nfs & done )
# within ~1-3 min: console shows
#   nfs - retry rename src to dst
#   Fatal trap 12 ... lockmgr_release+0x11  (kernel panic)

Success criterion (baseline): panic with the signature above within minutes. Patched: the retry message may still appear (it did, 8x), but the system stays up with no trap/panic (see run.log for the observed post-race umount caveat, which could not be compared against baseline).

DF-3061 — VERDICT

Status: reproduced (kernel panic, 2/2 runs) Impact: panic/dos — remote authorized-client-triggerable kernel crash; underlying mechanism includes leaked exclusive ncp locks, cache/cred refcount leaks, and double-drop (refcount-underflow / UAF) paths. Confidence: certain (structural root cause + two deterministic reproductions + faulting-instruction correlation)

Root cause (path:line)

1. sys/vfs/nfs/nfs_subs.c:1003-1011 — nfs_namei()'s "retry case" rescues only nd->nl_path from the reused nlookupdata. nlookup_init_raw() (called at nfs_subs.c:1155; implementation sys/kern/vfs_nlookup.c:255, note the unconditional bzero at :265) then wipes the rest of the previous iteration's live state without releasing it: * nl_nch — an EXCLUSIVELY LOCKED, held leaf ncp (NLC_NCPISLOCKED set by cache_lock4_tondlocked(), sys/kern/vfs_cache.c:1154, invoked from nfs_serv.c:2246), * nl_rootnch / nl_jailnch cache refs (set by nlookup_init_raw:280-282), * nl_cred — a crhold()'d ucred (nlookup_init_raw:283). The proper release protocol is nlookup_done() (vfs_nlookup.c:366-400), which is never run between iterations.

2. sys/vfs/nfs/nfs_serv.c:2255-2271 — nfsrv_rename()'s topology-revalidation retry (goto again) additionally leaves fnchd_status/tnchd_status == 2 and stale fdirp/tdirp pointers. If the SECOND iteration's nfs_namei() fails, the nfsmout cleanup (nfs_serv.c:2382-2391,2395) cache_unlocks and cache_drops the already-released handles and vreles the already-vrele'd tdirp — double-drop / refcount underflow / use-after-free.

Trigger

Any NFS client permitted by the server's exports issues RENAME(src,dst) RPCs racing concurrent modification (rename/unlink) of the same names — from a second client or a local process on the exported tree. When the namecache topology changes between the two nfs_namei() lookups and cache_lock4_tondlocked() revalidation, the retry branch fires (console marker: "nfs - retry rename src to dst").

Observed (stock INVARIANTS kernel, guest uname in env.txt)

Run 1 and Run 2 (identical signature, ~2-2.5 min of racing each):

nfs - retry rename src to dst Fatal user address access from kernel mode from nfsd at ffffffff806478a1 Fatal trap 12: page fault while in kernel mode fault virtual address = 0xc8 current process = 2145 / 839 (nfsd) Stopped at lockmgr_release+0x11: movq 0x8(%rdi),%rsi

fault address 0xc8 with rdi=0xc0 == offsetof(struct namecache, nc_lock) (sys/sys/namecache.h:125-148, nc_lock in its own __cachealign block) — i.e. a lock operation on an nchandle whose ncp is NULL/stale, exactly the corrupted-handle state the retry protocol leaves behind. panic.txt and panic_run2.txt hold both captures.

Exploit chain

Panic chain (proven): client RENAME race -> nfsrv_rename retry branch -> nfs_namei isretry bzero of live nd / stale fnchd_status/tdirp cleanup -> lock op on stale/NULL ncp -> fatal trap 12, all CPUs stopped. uid0 chain: not developed. The double-vrele/double-cache-drop components are refcount-underflow primitives on kernel objects (vnode, namecache), in principle exploitable with slab grooming; not pursued within this run's budget (the deterministic NULL-deref dominates the observable behavior).

Fix

fix.diff (two hunks): * nfs_subs.c: call nlookup_done(nd) in the retry path after rescuing the buffer (NLC_HASBUF cleared, nl_path NULLed first so the buffer survives) — releases the ncp lock, all cache refs and the cred. * nfs_serv.c: reset fnchd_status/tnchd_status to 0 and NULL fdirp/tdirp after their release in the retry branch, so the second iteration's error cleanup cannot double-release them.

Fix validation

Baseline (stock kernel): 2/2 panics within ~2.5 min of racing. Patched (make nativekernel with fix.diff applied): kernel rebuilt and booted; identical race run longer than 2x the baseline reproduction time — no panic, retry branch still exercised (kprintf seen), system stayed healthy. See run.log (which also records the verbatim build/install completion banners; build.log was lost to a failed scp before the guest reset).