DF-2997 / verdict.json
{ "finding_id": "DF-2997", "status": "reproduced", "reproduced": 1, "impact": "dos", "confidence": "certain", "verdict": "Reproduced end to end on the stock INVARIANTS guest with an in-guest userland NFSv3/UDP server. nfs_nget returns vnodes LK_EXCLUSIVE-locked (nfs_node.c:129 vget, :163+:216 getnewvnode/vx_downgrade); a LOOKUP or MKDIR reply carrying a valid filehandle followed by truncated post-op attributes makes nfsm_postop_attr / nfsm_mtofh return EBADRPC after a successful nfs_nget, and the nfs_lookup nfsmout tail (nfs_vnops.c:1272) and the nfs_mkdir error tails (:2348, :2361) drop the locked vnode with vrele() instead of vput(), so the exclusive lock is never released. Demonstrated consequences on stock: the triggering syscall sees a clean errno=72 while each occurrence permanently wedges one vnode; any later reference to the same name sleeps forever in vget/vn_lock (state D2, wchan vnode, kill -9 ignored - two unkillable processes left running); 82 leaked vnodes showed zero reclamation after 20s (672 forever); umount -f and even mount listing wedge in D-state; under kern.maxvnodes pressure the vnlru kernel recycler thread wedged permanently (B5 wchan vnode) and an innocent cron daemon wedged at exit; only a reboot recovers. No memory corruption and no info leak, so the ceiling is denial of service; precondition is attacker-controlled or spoofed NFS server replies (NFSv3/UDP, AUTH_SYS, no integrity) or an attacker-influenced export mounted by the client, with every triggering syscall running unprivileged (uid 1001). fix.diff (vput for nfs_nget-sourced newvp at the three sites, vrele kept for the dvp-echo cases, mirroring the in-tree idiom of nfs_nresolve:1083-1089) was built and booted: identical honest-server behavior, same errno=72 protocol errors, zero D-states, vnodes reclaimed (665 to 629 to 481), vnlru and cron healthy under identical pressure. One separate latent defect was unmasked: vflush skips v_type==VNON vnodes (vfs_mount.c:931/:978) so forced unmount of a mount holding never-attributed vnodes panics (unmount: dangling vnode); proven pre-existing on the STOCK kernel via the already-vput error path of nfs_mknodrpc with one truncated MKNOD reply (run_stocknod.raw, panic_stocknod.txt); on stock the DF-2997 wedge masks it by hanging umount forever instead - strictly worse.", "exploit_chain": "malicious/compromised/spoofing NFS server (or attacker-influenced export) replies LOOKUP/MKDIR with valid fh + truncated attrs -> unprivileged client user runs touch/mkdir -> nfs_nget creates vnode LK_EXCLUSIVE-locked -> EBADRPC error path vrele()s it -> vnode permanently locked with refcnt 0 -> any later lookup of that name (any API: nresolve, old lookup, mtofh) blocks forever in vget/vn_lock, uninterruptible and unkillable -> repeat with N distinct filehandles -> N permanently unreclaimable vnodes -> vnode-pool pressure drives the vnlru deactivate rover into vx_lock on a wedged vnode -> recycler kernel thread dies -> innocent processes wedge at exit via allocvnode_gc -> system-wide vnode exhaustion, no unmount, no recovery without reboot", "evidence": "run.raw.baseline (full stock log: 82x errno=72, NVN 599->672 no reclaim, probes D2 wchan vnode surviving kill -9, umount/mount D0); fakesrv.log.baseline (LOOKUP #1 NOENT nresolve, #2 truncated = the leak, #3 = wedge attempt); vnlru.after.baseline (vnlru B5 vnode wedged, cron D5Vs vnode); run_fixed2.raw (patched: probes fail fast and exit, NO_PROBE_PROCESSES, NVN 665->629->481, vnlru vlruwt, cron nanslp); run_stocknod.raw + panic_stocknod.txt (stock kernel panics unmount: dangling vnode via nfs_mknodrpc truncated-MKNOD path, no DF-2997 site involved - proves pre-existence); fix.diff (3 hunks, applied in guest, nativekernel rebuilt, validated)", "kernel_refs": [ "sys/vfs/nfs/nfs_node.c:129", "sys/vfs/nfs/nfs_node.c:216", "sys/vfs/nfs/nfs_vnops.c:1238", "sys/vfs/nfs/nfs_vnops.c:1272", "sys/vfs/nfs/nfs_vnops.c:2348", "sys/vfs/nfs/nfs_vnops.c:2361", "sys/vfs/nfs/nfsm_subs.c:412", "sys/vfs/nfs/nfsm_subs.c:493", "sys/kern/vfs_lock.c:589", "sys/kern/vfs_lock.c:380", "sys/kern/vfs_lock.c:890", "sys/kern/vfs_mount.c:548", "sys/kern/vfs_mount.c:931", "sys/kern/vfs_syscalls.c:1085", "sys/vfs/nfs/nfs_vnops.c:1083" ], "poc_changes": "No seeded pack existed; built the rig from scratch, adapted from the proven DF-2996 userland NFSv3/UDP fake server. Key design decisions verified against the DF client source: per-name LOOKUP counting (#1 = nresolve pre-check must get NOENT so the create proceeds, #2 = compat-shim VOP_OLD_LOOKUP gets the truncated positive reply that leaks, #3+ = wedge trigger); three exports keyed by root filehandle so one server instance serves control/lkmal/mkmal mounts simultaneously; deterministic name->fh mapping so repeats hash-hit the wedged vnode; a fourth export /nd truncating MKNOD replies to prove the umount panic pre-exists on stock through nfs_mknodrpc. Also fixed in the process: run.sh phase ordering (reclaim pressure sysctl moved before growth for the fixed run), binary pre-warming before kern.maxvnodes lowering (unwarmed execs wedge at exit on stock via allocvnode_gc - itself corroborating evidence), and probe.c gained stat and mknod ops plus an unistd include fix.", "attempts": 4, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 5100, "guest_dirty": 0, "build_cmd": "cd /root/df2997 && sh build.sh (cc -O2 -Wall -o fakesrv fakesrv.c; cc -O2 -Wall -o probe probe.c); fix kernel: cd /usr/src && git apply fix.diff && make nativekernel KERNCONF=X86_64_GENERIC && make installkernel KERNCONF=X86_64_GENERIC", "run_cmd": "sh run.sh (stock baseline); sh run_fixed2.sh (patched validation); sh run_stocknod.sh (stock umount-panic pre-existence proof)", "code_hash": "c581534b2515c9b839d208876b8f55fc806c8383b52224924ac5df5bc45905c7", "fix_status": "fixed", "fix_verdict": "fix.diff applied to the guest /usr/src (git apply, 3 hunks clean), kernel rebuilt with make nativekernel KERNCONF=X86_64_GENERIC and installed (build in build.log in guest), rebooted into kernel #1. Identical rig: all wedge probes fail fast and exit (no D-state, no unkillable processes), leaked vnodes are reclaimed (665->629->481 under the same maxvnodes pressure that stock never reclaims under), vnlru stays vlruwt, cron survives, honest control mount bit-identical. The errno=72 EBADRPC user-visible error is unchanged, as expected. Remaining caveat, proven pre-existing on stock: umount -f of a still-poisoned mount panics (unmount: dangling vnode) because vflush skips VNON vnodes - recommend a follow-up fix in vmntvnodescan/vflush or a full vgone teardown of never-attributed vnodes in the NFS error paths.", "notes": "Team directive asked to close or flag the privesc question per facts: there is no privilege-escalation primitive here - the bug is a lock/lifetime defect producing permanent uninterruptible sleeps and resource exhaustion only; classified dos. The unmasked VNON/vflush umount panic is a separate latent bug (stock-reachable via nfs_mknodrpc truncated reply, one unprivileged mknod FIFO + umount -f); flagged for follow-up. Baseline guest had to be force-reset (wedged D-state processes made clean shutdown impossible - itself evidence). Final guest state reset to clean with-src snapshot.", "recommended_fix": "vput() nfs_nget-sourced vnodes at the three error tails (nfs_vnops.c:1272, :2348, :2361), keeping vrele() only for the dvp-echo cases - mirrors nfs_nresolve:1083-1089; follow-up: make vflush handle VNON vnodes so never-attributed vnodes cannot dangle at forced unmount" } |