nfs_mount() swallows nfs_mountroot() failure: failed diskless NFS-root mount reported as success -> NULL mnt_data deref (Fatal trap 12) in nfs_root instead of clean 'Root mount failed' fallback
Summary
nfs_mount discards nfs_mountroot's error and returns 0, so vfs_mountroot_try's success branch VFS_ROOTs a never-mounted mount: nfs_root computes nmp=mp->mnt_data==NULL and lwkt_gettoken(&nmp->nm_token) faults at fixed VA 0x450 then re-fault loops in DDB; the unwind also double-drops vfc_refcount. vkernel-rig-reproduced (unreachable NFS root -> 'mount ... : 65' then Fatal trap 12 + 4.2M-line spin); the shared nfs_vfsops code means pc64 PXE clients hit the same panic when the NFS server is unreachable (on-path attacker converts transient outage into a panic loop). Fixed-offset NULL deref, DoS ceiling. Error-propagation + refcount fix validated (clean 'Root mount failed' + mountroot> fallback).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-5534 Β· 15 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 3.2 KB | β raw | |
| VERDICT.md | β | 4.5 KB | β raw | |
| trigger.sh | β | 1.5 KB | view raw | |
| matrix.sh | β | 2.6 KB | view raw | |
| build.sh | β | 118 B | view raw | |
| run.sh | β | 90 B | view raw | |
| build.log | β | 488 B | view raw | |
| run.log | β | 2.9 KB | view raw | |
| run.crashloop_tail.log | β | 2.6 KB | view raw | |
| run.fixed.log | β | 1.9 KB | view raw | |
| panic.txt | β | 1.8 KB | view raw | |
| env.txt | β | 810 B | view raw | |
| fix.diff | β | 602 B | view raw | |
| manifest.json | β | 1.1 KB | view raw | |
| verdict.json | β | 4.1 KB | view raw |
DF-5534 β nfs_mount() swallows nfs_mountroot() failure β NULL mnt_data deref in nfs_root on failed diskless NFS-root mount
What
nfs_mount() discards the return value of nfs_mountroot() and unconditionally
returns 0 (success) to the VFS layer:
sys/vfs/nfs/nfs_vfsops.c:942
if (path == NULL) {
nfs_mountroot(mp);
return (0); /* error swallowed */
}
When the diskless NFS root mount fails (unreachable server, failed mount RPC),
vfs_mountroot_try() therefore takes the SUCCESS branch
(sys/kern/vfs_conf.c:441 error = VFS_MOUNT(...) β if (!error) break;) and
runs the success path on a mount that was never mounted: mountlist_insert(),
inittodr(), then VFS_ROOT(mp) β nfs_root():
sys/vfs/nfs/nfs_vfsops.c:~
nmp = VFSTONFS(mp); /* mp->mnt_data == NULL (mountnfs never ran) */
lwkt_gettoken(&nmp->nm_token); /* &NULL->nm_token == 0x450 β Fatal trap 12 */
The error unwind also double-drops vfc_refcount
(nfs_mountdiskless haderror nfs_vfsops.c:772 + nfs_mountroot nfs_vfsops.c:660)
for the single reference taken by vfs_rootmountalloc() (vfs_mount.c:344).
Reachability (this file's surface)
sys/platform/vkernel64/x86_64/autoconf.c pxe_setup_nfsdiskless()
(autoconf.c:388-486) parses operator command-line kenv (-e boot.netif.ip,
boot.netif.netmask, boot.netif.name, boot.nfsroot.server) and sets
nfs_diskless_valid = 1 (autoconf.c:485); cpu_rootconf() then sets
rootdevnames[0] = "nfs:" (autoconf.c:279). With the NFS server unreachable,
mountroot fails β panic loop instead of the intended "Root mount failed" +
manual-prompt fallback. The same nfs_vfsops.c code is shared by pc64
(loader/PXE-supplied boot.netif.* env); the vkernel is the reproducible rig.
Reproduce (guest, root; vkernel rig)
sh build.sh # VKERNEL64 kernel + /vkroot.img (see env.txt) sh run.sh # boots vkernel with hostile NFS-root boot args
Success criterion (baseline): console shows
nfs_mountroot: mount 127.0.0.1: on /: 65 immediately followed by
Fatal trap 12: page fault while in kernel mode (fault VA 0x450,
IP lwkt_gettoken+0x64), then a db> crash-spin loop (millions of re-faults).
Patched: Root mount failed: 65 β manual mountroot> prompt β
ufs:vkd0 mounts β boots to userland, zero traps (run.fixed.log).
Fix validation
fix.diff (error propagation + removal of the extra refcount drop):
baseline kernel MD5 a3a76c8c358d3df60cea81a31587ccf6 crashes as above;
patched kernel MD5 3cb21e35f1447c8d69ceb1401226fefee survives and boots.
Bad behavior gone. See VERDICT.md.
Files
- trigger.sh exact repro (deepX boot args) + nbuf control boot
- matrix.sh wider negative-validation matrix (parse-garbage boots)
- build.sh guest kernel/image build commands
- run.sh minimal single-run driver
- build.log baseline + patched kernel build tails
- run.log baseline decisive console (crash)
- run.fixed.log patched full console (57 lines, 0 traps, boots to userland)
- panic.txt Fatal trap 12 excerpt (baseline)
- env.txt uname, kernel MD5s, rig details
- fix.diff git-apply-able fix
- manifest.json machine catalog
DF-5534 VERDICT β reproduced (panic), fix validated
Result
| kernel | behavior under identical boot args |
|---|---|
baseline a3a76c8c358d3df60cea81a31587ccf6 |
nfs_mountroot: mount 127.0.0.1: on /: 65 β immediately Fatal trap 12 (VA 0x450, lwkt_gettoken+0x64), then an unbounded db> re-fault loop (4,186,854 console lines) |
patched 3cb21e35f1447c8d69ceb1401226fefee |
Root mount failed: 65 β manual mountroot> prompt β ufs:vkd0 mounts β boots to userland; 0 traps, 57-line console, vkernel alive |
Root cause (path:line)
- Operator kenv (vkernel
-e, pc64 loader/PXE) βpxe_setup_nfsdiskless()parsesboot.netif.*/boot.nfsroot.*(all writes bounded; sys/platform/vkernel64/x86_64/autoconf.c:388-486) and setsnfs_diskless_valid = 1(:485).cpu_rootconf()setsrootdevnames[0] = "nfs:"(:279). vfs_mountroot_try()allocates the root mount (onevfc_refcount++, sys/kern/vfs_mount.c:344;mnt_data == NULLviaM_ZERO, :340) and callsVFS_MOUNT(mp, NULL, NULL, ...)(sys/kern/vfs_conf.c:441).nfs_mount()seespath == NULLβnfs_mountroot(mp); return (0);(sys/vfs/nfs/nfs_vfsops.c:942-945 β the error is discarded).- NFS server unreachable β
md_mountRPC fails 65 βnfs_mountdisklesshaderror:mp->mnt_vfc->vfc_refcount--(:772) βnfs_mountrooterror path:vfc_refcount--again (:660; double drop of the single alloc reference) β returns 65 β into the void. nfs_mountreturns 0 βvfs_mountroot_trytakes the success branch (vfs_conf.c:441-476):mountlist_insert(mp), thenVFS_ROOT(mp, &vp).nfs_root():nmp = VFSTONFS(mp)=mp->mnt_data= NULL (mountnfs()never ran;md_mountfailed first) βlwkt_gettoken(&nmp->nm_token)β read at 0x450 =offsetof(struct nfsmount, nm_token)β Fatal trap 12, swapper, boot CPU. Every ddb continue re-faults (lwkt_relalltokensat 0x458) β panic-spin loop. The intended behavior ("Root mount failed", manual prompt fallback, as observed patched) never runs.
Why it matters
- Any vkernel booted with NFS-root boot args whose server is unreachable panic-loops at boot instead of failing cleanly (operator/guest-of-guest DoS; the vkernel process spin-crashes in ddb).
- Same shared nfs_vfsops.c code on pc64: an NFS-root machine whose server is down/unreachable at boot panics in the mountroot path (remote-triggerable boot DoS: an on-path attacker who blackholes the server port converts a transient outage into a kernel panic loop). NULL-deref only β fixed offset, not attacker-controlled data β so impact ceiling is DoS, not corruption.
Not this bug
- The
boot.netif.*/boot.nfsroot.*parsing itself (this file) is bounded and survived every hostile-input boot in matrix.sh: overflow%dfields, unbounded%xruns, 90-char path truncated byte-exact at MNAMELEN-1=87 with clean NUL termination, 20-byte in-bounds nfshandle both failure and success (closing-X) branches,kern.nbuf=99999999999safely clamped ("Warning: nbufs capped at 2097104 for valloc considerations", autoconf.c:156-160). - DF-1101 (decode_nfshandle one-past write into root_saddr.sin_len) is a separate, already-filed finding; not re-verified here.
Fix
fix.diff: propagate the error (return (nfs_mountroot(mp));) and drop the
redundant vfc_refcount-- at nfs_vfsops.c:660 so exactly one reference drop
(the one in nfs_mountdiskless haderror) pairs the vfs_rootmountalloc
reference. Validated on the guest: bad behavior gone, clean fallback + boot.
Honesty notes
- The guest-side baseline crash console was overwritten by the fix-validation run (run_vk reuses the console filename); the decisive baseline evidence is preserved in run.log (filtered), run.crashloop_tail.log (raw tail of the 4.2M-line crash loop) and panic.txt (verbatim trap blocks incl. the first 0x450 fault). The patched run.fixed.log is complete and untrimmed.
- RUN A of matrix.sh (same class, no closing-X handle) spent its whole 120s window in RPC retransmit ("RPC timeout for server 127.0.0.1") and never reached the crash within the window; RUN E (deepX, TCP connect refused fast) reproduced deterministically. Both are the same unwind.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedBaseline vkernel (kernel.debug MD5 a3a76c8c358d3df60cea81a31587ccf6): Fatal trap 12 at lwkt_gettoken+0x64, fault VA 0x450 (NULL nfsmount via mnt_data) immediately after 'nfs_mountroot: mount 127.0.0.1: on /: 65', then unbounded db> re-fault loop (4,186,854 console lines). Patched vkernel (fix.diff applied in guest /usr/src, rebuilt, MD5 3cb21e35f1447c8d69ceb1401226fefee): identical boot args -> 'Root mount failed: 65', auto-retry fails cleanly ('no disk named vkd0s1a'/'Root mount failed: 6'), manual mountroot> prompt accepts 'ufs:vkd0', devfs+ufs mount and the system boots to userland; 0 traps, vkernel alive (run.fixed.log). Bad behavior gone.
fix.diff, run.log, run.crashloop_tail.log, panic.txt, run.fixed.log, build.log
Confirmed kernel references
Detail
Evidence (decisive lines)
run.log (baseline: 'mount ... : 65' then Fatal trap 12 x2), run.crashloop_tail.log + panic.txt (trap blocks VA 0x450/0x458, IP lwkt_gettoken+0x64 / lwkt_relalltokens+0x3c, nm symbolization), run.fixed.log (patched: 'Root mount failed: 65', mountroot> prompt, ufs:vkd0 boots, 0 traps, 57-line console), build.log (baseline a3a76c8c... / patched 3cb21e35... kernel MD5s), fix.diff
PoC changes
Seed-less (fresh pass-2 finding). Rig adapted from DF-5337 proven vkernel rig; trigger boots the vkernel with in-bounds hostile boot.netif./boot.nfsroot. args pointing at 127.0.0.1 (fast TCP refuse -> md_mount error 65).
Verified recommended fix
Propagate nfs_mountroot() failure out of nfs_mount() (return the error) and remove the duplicate vfc_refcount-- at nfs_vfsops.c:660 so exactly one drop pairs vfs_rootmountalloc().
Verdict
nfs_mount() discards nfs_mountroot()'s error and returns 0 (nfs_vfsops.c:942-945), so a failed diskless NFS-root mount is reported to vfs_mountroot_try as success; the success path then calls VFS_ROOT on a never-mounted mp whose mnt_data is NULL and nfs_root executes lwkt_gettoken(&nmp->nm_token) at 0x450 -> Fatal trap 12 panic-spin loop at boot instead of the intended 'Root mount failed' + manual prompt fallback. Triggered from vkernel command-line boot args via pxe_setup_nfsdiskless (autoconf.c:388-486, nfs_diskless_valid=1 at :485, rootdevnames[0]='nfs:' at :279) with an unreachable NFS server; shared nfs_vfsops.c code means pc64 NFS-root clients hit the same panic when the server is unreachable. NULL-deref only (fixed struct offset) - DoS ceiling, no corruption. Fix (error propagation + single refcount drop) validated on the guest: patched kernel prints 'Root mount failed: 65', falls back to the manual mountroot prompt, boots ufs:vkd0 to userland with zero traps.
No comments yet.