β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-5831

nlookup() retry path NULL-dereferences nd->nl_basench for nlookup_init_raw()/_root()-initialized nd β€” every NFS-server name lookup that hits a retry (concurrent rename, ESTALE, destroyed parent) panics the kernel in nfsd

Summary

The doretry block resets to the base directory with cache_copy(nd->nl_basench, ...) but nl_basench is only initialized by nlookup_init/init_at; nlookup_init_raw and nlookup_init_root bzero the nd and never set it. nfs_namei β€” the NFS server's name resolution for every (dir-fh,name) RPC (LOOKUP/CREATE/REMOVE/RENAME/MKDIR/LINK/SYMLINK/SETATTR-by-name) β€” and the WebNFS public-handle lookup both use init_raw. Any doretry dereferences NULL at cache_copy's first nch->ncp load. Guest-reproduced: loopback NFS rig, thread A flooding lstat RPCs inside an exported dir while thread B ping-pong renames it β€” 'Fatal trap 12 fault virtual address = 0x0, Stopped at cache_copy+0x11', current process = nfsd, guest down within 2 minutes. Remote panic of the NFS server by any export-permitted client, no auth beyond the export ACL; also a reliability crash for ordinary concurrent NFS workloads. NFS not enabled by default -> High not Critical. Private ref-held nl_basench_store fix validated by guest kernel rebuild + 2x300s identical rig runs with zero panics (baseline panicked in <=2 min).

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-5831 Β· 16 files
FileTypeDescriptionSize
hammer.c β€” 1.5 KB view raw
renamer.c β€” 969 B view raw
run.sh β€” 162 B view raw
build.sh β€” 130 B view raw
panic.txt β€” 863 B view raw
serial_console_baseline.log β€” 2.1 KB view raw
run.log β€” 582 B view raw
run.2.log β€” 498 B view raw
fix_run.log β€” 582 B view raw
build.log β€” 5.1 MB ↓ download
fix_build.log β€” 5.1 MB ↓ download
fix.diff β€” 2.4 KB view raw
env.txt β€” 452 B view raw
README.md β€” 3.9 KB ↓ raw
VERDICT.md β€” 4.6 KB ↓ raw
verdict.json β€” 6.0 KB view raw

DF-5831 β€” nlookup() retry path NULL-derefs nd->nl_basench for nlookup_init_raw() nd's β†’ remote NFS-server kernel panic

What

nlookup()'s internal retry path (added for autofs mounts / NFS ESTALE / namecache generation changes) resets the lookup to the base directory:

    sys/kern/vfs_nlookup.c:1367-1385
    if (doretry && !inretry) {
        ...
        cache_drop(&nd->nl_nch);
        cache_copy(nd->nl_basench, &nd->nl_nch);   /* :1380 */
        ...
        goto nlookup_start;
    }

nl_basench is only ever initialized by nlookup_init() (vfs_nlookup.c:149/159/170) and nlookup_init_at() (:225/228/239). nlookup_init_raw() (:255-291) and nlookup_init_root() (:297-335) leave it NULL (the whole nd is bzero()'d).

Any doretry during an nlookup on an init_raw/_root-initialized nd therefore executes cache_copy(NULL, &nd->nl_nch):

    sys/kern/vfs_cache.c:996-1000
    void cache_copy(struct nchandle *nch, struct nchandle *target)
    {
        ...
        ncp = nch->ncp;     /* NULL deref, ncp is at offset 0 */

β†’ Fatal trap 12: page fault on read at 0x0 in cache_copy+0x11.

Reachability

nlookup_init_raw() is called from:

  • nfs_namei() (sys/vfs/nfs/nfs_subs.c:1155) β€” the NFS server's name resolution for every RPC carrying a (dir filehandle, name) pair: LOOKUP, CREATE, REMOVE, RENAME (both sides), MKDIR, LINK, SYMLINK, SETATTR/GETATTR-by-name, etc.
  • nfsrv3_public (sys/vfs/nfs/nfs_serv.c:568) β€” the WebNFS public filehandle index lookup.
  • vfs_mountroot_devfs() (sys/kern/vfs_conf.c:238) β€” boot only.

nlookup_init_root() β€” vfs_synth only (boot).

The doretry triggers, all raceable by an NFS client against concurrent modification of the exported tree (a second client, a second thread of the same client, or a local process):

  • generation change of the start directory or any component, detected at vfs_nlookup.c:1148-1158 / :1313-1323 β€” e.g. a concurrent RENAME sub -> sub2 of the directory being searched bumps its nc_generation (_cache_unlink_parent + _cache_link_parent both _cache_ncp_gen_enter/exit, vfs_cache.c:783/840, 2155);
  • parent destroyed (NCF_DESTROYED) during the leaf lookup, vfs_nlookup.c:1229-1232 β€” rename-over of the directory;
  • ESTALE from cache_resolve (vfs_nlookup.c:802-806, :882-886);
  • the trailing-./.. EINVAL retry (vfs_nlookup.c:848-852).

Impact: kernel panic in the nfsd process β€” remote denial of service of the NFS server by any client permitted by the exports (no authentication beyond the export ACL); also reachable by ordinary concurrent NFS workloads (lookups racing renames of the containing directory) without hostile intent. Not enabled by default (requires the NFS server to be running), hence High rather than Critical.

Files

  • hammer.c β€” client-side LOOKUP hammer: stat("/mnt/nfs/sub/zzzNNN") loop β†’ server-side nfsrv_lookup β†’ nfs_namei β†’ nlookup with nl_basench == NULL.
  • renamer.c β€” renames sub <-> sub2 (locally on the export and over the NFS mount) to churn sub's nc_generation.
  • run.sh β€” sets up the loopback NFS server + client, runs the rig.
  • panic.txt β€” captured panic (see below).
  • fix.diff β€” verified fix.
  • build.log, run.log, run.2.log β€” fix-validation kernel build and the two post-fix survival runs.

Reproduce

In the guest (as root):

sh /root/poc/DF-5831/run.sh

Expected on a vulnerable kernel (observed ≀ 2 minutes into the run):

kern_rename: retry due to race on: "sub" -> "sub2" (16,0)
Fatal user address access from kernel mode from nfsd at ffffffff806dac71
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x0
...
Stopped at      cache_copy+0x11:        movq    (%rdi),%r13

current process is the nfsd; the fault is a read of NULL->ncp (ncp is the first member of struct nchandle), i.e. exactly cache_copy(nd->nl_basench, ...) with nd->nl_basench == NULL.

VERDICT.md
↓ download raw

DF-5831 VERDICT β€” REPRODUCED (kernel panic, remotely triggerable via NFS)

Bottom line

Reproduced on the first guest run, in under two minutes of racing. The stock INVARIANTS guest (DragonFly 6.5-DEVELOPMENT, see env.txt) panicked in the nfsd process with exactly the predicted signature:

kern_rename: retry due to race on: "sub" -> "sub2" (16,0)
Fatal user address access from kernel mode from nfsd at ffffffff806dac71
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x0
...
Stopped at      cache_copy+0x11:        movq    (%rdi),%r13

(panic.txt β€” captured from the serial console; the run started at 10:43 UTC, the ssh session died and the guest was down by 10:46 UTC.)

Why this is the bug I claimed (not some other crash)

  1. Stopped at cache_copy+0x11: movq (%rdi),%r13 β€” cache_copy's first instruction sequence loads nch->ncp (ncp is the first member of struct nchandle, sys/sys/namecache.h); the fault is a read at virtual address 0x0 with %rdi == NULL, i.e. the first argument of cache_copy() was NULL.
  2. cache_copy(nd->nl_basench, ...) is called with a NULL first argument from exactly one site reachable in the nfsd process: sys/kern/vfs_nlookup.c:1380 (nlookup's doretry reset block). The other four call sites (vfs_nlookup.c:150, 160, 171, 242) pass &fd_ncdir / &fd_nrdir / &fp->f_nchandle β€” never NULL.
  3. nfs_namei() (sys/vfs/nfs/nfs_subs.c:1155) initializes its nd with nlookup_init_raw(), which bzero()s the whole structure and never sets nl_basench (vfs_nlookup.c:265 vs. :280-290). Every server-side name RPC (LOOKUP/CREATE/REMOVE/RENAME/...) goes through it.
  4. The doretry fired on a namecache generation change: the console shows kern_rename: retry due to race on: "sub" -> "sub2" immediately before the trap β€” my racer was renaming the looked-up directory sub <-> sub2, and _cache_unlink_parent() + _cache_link_parent() both _cache_ncp_gen_enter/exit() the renamed ncp (sys/kern/ vfs_cache.c:783/840, 2155), which nlookup() detects at vfs_nlookup.c:1313-1332 and turns into doretry = TRUE.
  5. Current process = nfsd (pid 1025) β€” the NFS server thread, i.e. the crash was driven by client RPCs racing a concurrent rename, not by the local racer's own path (a local rename uses nlookup_init() and has a valid nl_basench).

Exploit chain / impact

Remote denial of service (kernel panic) of an NFS server by any client permitted by the exports: hammer LOOKUP RPCs on names inside a directory while the same (or another) client renames that directory. No authentication beyond the export ACL. Also reachable as a reliability crash by ordinary concurrent NFS workloads. The fault is a deterministic NULL deref β€” no memory-corruption primitive, no uid0 route (the guest is an intentionally permissive target, but NULL page mapping is not attacker-controllable here; this is panic-class only).

Fix validation

fix.diff (git-apply-able against the audit tree; applied with patch -p1 inside the guest's /usr/src):

  • nlookup_init_raw() / nlookup_init_root() now store a private, ref-held copy of the start handle in a new nl_basench_store field of struct nlookupdata and point nl_basench at it;
  • nlookup_done() drops the store.

Baseline (stock kernel): panic in nfsd ≀ 2 minutes into the rig (panic.txt). Patched (kernel rebuilt with make nativekernel KERNCONF=X86_64_GENERIC, rebooted): the identical rig ran to completion twice β€” full 300 s race window each, racers verified running, mount | grep nfs verified, no panic, guest stayed up, script printed NO PANIC (run.log, run.2.log). The previously-observed bad behavior is gone.

Kernel references

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

fix.diff applied to the guest's /usr/src (patch -p1, 4 hunks clean), kernel rebuilt with make nativekernel KERNCONF=X86_64_GENERIC (build.log, RC=0), installed and rebooted. The identical PoC rig then ran the full 300-second race window twice (run.log, run.2.log) with hammer/renamer processes verified running and the NFS mount verified active - no panic either time, guest stayed up, dmesg clean. Baseline had panicked in under 2 minutes. The fix is confirmed to remove the crash without changing any other behavior observed.

fix.diff; fix_build.log (=build.log, full untrimmed 14897-line kernel build ending 'Kernel build for X86_64_GENERIC completed ... RC=0'); fix_run.log & run.2.log (both end 'NO PANIC'); panic.txt (baseline)
↓ fix.diffper-fix-DF-5831

Confirmed kernel references

Detail

Exploit chain

1) Attacker is any NFS client permitted by the server's exports (no auth beyond the export ACL). 2) Thread A issues LOOKUP-bearing RPCs (stat/open) on names inside an exported directory D (each becomes a server-side nfsrv_lookup -> nfs_namei -> nlookup_init_raw -> nlookup with nl_basench==NULL). 3) Thread B (second connection/client or local process) renames D <-> D2 in a loop; each rename bumps D's ncp nc_generation via _cache_unlink_parent/_cache_link_parent (_cache_ncp_gen_enter/exit, vfs_cache.c:783/840/2155). 4) Any A-lookup in flight across a B-rename detects the generation change at vfs_nlookup.c:1313, sets doretry, and the retry block runs cache_copy(NULL, &nd->nl_nch) at :1380. 5) Fatal trap 12 read at 0x0 in cache_copy+0x11 inside the nfsd process - kernel panic, NFS server (and the whole machine) down. Observed in <=2 minutes of hammering; ordinary concurrent NFS workloads reach the same race without hostile intent.

Evidence (decisive lines)

panic.txt (full console panic: 'Fatal user address access from kernel mode from nfsd', 'fault virtual address = 0x0', 'Stopped at cache_copy+0x11: movq (%rdi),%r13'); serial_console_baseline.log (console context incl. the preceding 'kern_rename: retry due to race on: sub -> sub2'); run.log + run.2.log (post-fix survival runs, 'NO PANIC', 300s each); build.log/fix_build.log (kernel build with fix.diff, RC=0); fix.diff (validated fix)

PoC changes

PoC authored fresh for this finding (no seed): hammer.c (NFS client LOOKUP hammer via lstat of random names under the mounted export's 'sub' dir), renamer.c (sub<->sub2 ping-pong, run both locally on the export and over the NFS mount), run.sh (loopback NFS server+client setup adapted from the DF-3061 rig, 8 nfsd threads, 300s race window). All syscall usage compiled and ran first-try on the guest; MD5s verified after copy.

Verified recommended fix

Give nlookup_init_raw()/nlookup_init_root() a private, ref-held retry base: add 'struct nchandle nl_basench_store' to struct nlookupdata, copy ncstart into it and point nl_basench at it in both initializers, and drop it in nlookup_done() (see fix.diff; validated in-guest).

Verdict

nlookup()'s internal doretry path (sys/kern/vfs_nlookup.c:1380) executes cache_copy(nd->nl_basench, &nd->nl_nch) with nl_basench == NULL whenever the nd was initialized by nlookup_init_raw()/nlookup_init_root() (vfs_nlookup.c:265 vs :280-290/:325-334), because those initializers never set nl_basench. Every NFS-server name RPC resolves through nfs_namei() (nfs_subs.c:1155) which uses nlookup_init_raw(), so any retry trigger during a server-side name lookup - a concurrent rename bumping the ncp nc_generation (detected at vfs_nlookup.c:1313-1332), ESTALE, or a destroyed parent (:1229-1232) - NULL-derefs at cache_copy (ncp is offset 0 of struct nchandle; fault read at 0x0). Reproduced on the stock INVARIANTS guest on the FIRST run, in under 2 minutes: 'Fatal trap 12: page fault while in kernel mode, fault virtual address = 0x0', 'Stopped at cache_copy+0x11: movq (%rdi),%r13', current process = nfsd, immediately after the console showed a rename race on the looked-up directory. Impact is a remote kernel panic of the NFS server by any client permitted by the exports (lookups racing renames of the containing directory); deterministic NULL deref, no memory-corruption primitive, so panic/DoS class only. Fix (nl_basench_store field holding a private ref'd retry base, set by init_raw/init_root, dropped by nlookup_done) validated in-guest: kernel rebuilt with make nativekernel, rebooted, identical rig run twice for the full 300s window each with racers verified running - NO PANIC both times, guest stayed up.