DF-3169 — autofs umount -f vs in-flight trigger/mkdir: use-after-free

Bug: An in-flight autofs vop (autofs_nresolve trigger path / autofs_readdir trigger path / autofs_nmkdir) holds only raw anp/amp pointers across multi-second daemon-dispatch sleeps and the 1–2 s retry-gap sleeps of autofs_trigger() (sys/vfs/autofs/autofs.c:427-455). autofs_unmount() (sys/vfs/autofs/autofs_vfsops.c:195-261) — reachable concurrently via umount -f (MNT_FORCE) — force-closes even referenced vnodes (sys/kern/vfs_mount.c:1165-1181), drains sc_requests with a 1 s-cadence scan, and then frees every autofs_node and the autofs_mount. Its "no new triggerings" assumption (autofs_vfsops.c:209-212) is false: a trigger thread parked in a retry gap inserts new requests — and dereferences/writes the freed node — after the free.

Sinks (all demonstrated or code-traced): - read-after-free: anp->an_parent, anp->an_mount, amp->am_* copied into the daemon-visible request (autofs.c:312-356) — leaked heap bytes of whichever kernel object reuses the 4288-byte autofs_mount block; - write-after-free: anp->an_retries++ (autofs.c:441-445), anp->an_cached/anp->an_wildcards (autofs.c:410-411), callout_reset(&anp->an_callout, …) arming a 600 s callout inside the freed 256-byte node block (autofs.c:412-413); - post-free request insertion with dangling ar_mount (autofs.c:345,362); - autofs_nmkdir NULL-deref: after force-vgone, dvp->v_mount == &dummymount ⇒ amp == dummymount.mnt_data == NULL ⇒ mtx_lock_ex_quick(&amp->am_lock) faults (autofs_vnops.c:251,263).

Trigger model: unprivileged churn (stat/ls under the automount — the trigger side) racing root's umount -f (or automount -u) — i.e. routine autofs admin operations on a box where users hit the automount.

Reproduce (deterministic KLD harness — primary proof)

Prerequisites (root, once)

kldload autofs
mkdir -p /df3169_mp
mount_autofs -f df3169 "" /df3169_mp     # run.sh sets the sysctls

Build

./build.sh      # builds df3169_race.ko in-guest (needs /usr/src)

Run (baseline, unpatched)

./baseline_run.sh   # == run.sh content; see run.log for the result

Expected (see run.log, reproduced verbatim on stock INVARIANTS kernel):

DF3169: dounmount returned 0 (ticks=15842) -- anp=... amp=... ARE FREED NOW
DF3169: reuse blocks: amp 0xfffff8008d4be800 ... SAME-BLOCK, node ... SAME-BLOCK
DF3169: DANGLING REQUEST ar=... id=1 ar_mount=0xfffff8008d4be800 == FREED amp
DF3169:   ar_from="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"...   <- read-after-free
DF3169 POLL: freed anp=... an_retries: N -> N+1 ...  <== WRITE-AFTER-FREE   (×10)
DF3169 W: trigger returned 60 (ticks=19042, dounmount_ticks=15842)

The harness re-allocates the freed blocks (marker 0x41 in the amp, zeroed objcache node) right after dounmount() returns — same CPU, same magazine — so the stale trigger's reads and writes become visible.

Run (production-shaped, probabilistic)

cc -O2 -pthread -o fake_daemon fake_daemon.c   # + autofs_ioctl.h
./variantB.sh            # 40 × (mount_autofs → trigger+mkdir storm → umount -f)

On the unpatched kernel this exercises the real userspace paths (58 daemon requests handled in the recorded run); the µs-wide autofs_nmkdir NULL-deref window did not hit within 40 cycles (run_variantB.log) — the deterministic KLD run is the proof.

Run (patched, fix.diff applied, kernel rebuilt)

Same baseline_run.sh; expected DIFFERENCE:

DF3169 W: trigger returned ... (ticks=..., dounmount_ticks=<same or later>)   # ordering flipped
DF3169: SUMMARY ... dangling=0                                                # no post-free request
# and NO "WRITE-AFTER-FREE" POLL lines

Fix

fix.diff (v2) — per-mount lifetime references: struct autofs_mount gains am_dying/am_triggers; autofs_nresolve/autofs_readdir/ autofs_nmkdir acquire a trigger reference for their whole body (ENXIO if dying / force-reclaimed); autofs_unmount() marks the mount dying before the request-drain loop and waits (bounded, EBUSY abort) for in-flight triggers before freeing any node. No mnt_lock is taken inside the vops — a first fix revision that used vfs_busy() was found to deadlock (ncp-lock → mnt_lock vs dounmount's mnt_lock → cache_inval ncp-lock; see FIX_NOTES.md / VERDICT.md) and was replaced.

DF-3169 VERDICT — autofs umount -f vs in-flight trigger/mkdir UAF

Status: REPRODUCED (deterministic KLD harness on stock INVARIANTS kernel; production-shaped userspace variant exercised 40 cycles). Impact: kernel use-after-free (read+write) + NULL-deref panic variant (memcorrupt bucket). Not a standalone unpriv→root chain: the free side requires umount -f (root), the trigger side is any unprivileged stat/ls under the automount — the realistic incident is root/cron (automount -u) racing user churn, producing silent heap corruption or a panic.

Root cause (path:line)

1. autofs_trigger() retry loop parks the calling thread with sc_lock dropped for autofs_retry_delay seconds between attempts, holding no reference of any kind on the mount (sys/vfs/autofs/autofs.c:451-453). The daemon-dispatch cv wait is equally unprotected but does keep a request on the TAILQ (which the unmount drains); the retry gap keeps nothing. The pre-trigger section of autofs_trigger_vn() (test_fs_root + nlookup_fs_root, sys/vfs/autofs/autofs_vnops.c:155-165) is a second, narrower gap.
2. autofs_unmount() assumes "All vnodes are gone … no new triggerings" (sys/vfs/autofs/autofs_vfsops.c:209-212). Under MNT_FORCE that is false: vflush(FORCECLOSE) force-vgones vnodes even with outstanding references (sys/kern/vfs_mount.c:1165-1181) and re-homes them to dummymount, so the in-flight vop keeps running; the 1 s-cadence request scan (autofs_vfsops.c:213-237) then observes an empty TAILQ during a trigger's retry gap and proceeds to free every autofs_node (incl. objcache_put) and the autofs_mount (kfree) (autofs_vfsops.c:239-258).
3. The parked thread wakes and its next autofs_trigger_one() call runs wholly on freed memory: - reads anp->an_parent/an_mount, autofs_path() walks the freed mount (autofs.c:312-322); - copies amp->am_from/am_prefix/am_options into the new request (autofs.c:351-356) — reads from the freed/reused 4288-byte block; - inserts that request with ar_mount = freed amp (autofs.c:345,362); - on each failure anp->an_retries++ (autofs.c:441-445); - if the daemon answers success it also writes anp->an_cached/an_wildcards and arms callout_reset(&anp->an_callout, autofs_cache*hz, …) — a 600 s callout installed in freed memory (autofs.c:409-413).
4. autofs_nmkdir() variant: after force-vgone dvp->v_mount == &dummymount, so amp = VFSTOAUTOFS(dvp->v_mount) is dummymount.mnt_data == NULL and mtx_lock_ex_quick(&amp->am_lock) (autofs_vnops.c:251,263) faults — guaranteed NULL-offset panic once the interleaving hits (µs window; not hit in 40 probabilistic cycles, code-trace certain).

Reproduction — baseline (run.log, decisive lines)

Stock X86_64_GENERIC (INVARIANTS) DragonFly 6.5-DEVELOPMENT #0. Harness df3169_race.ko: kthread W in the REAL autofs_trigger() on the mount's root node (sysctls: timeout=1, retry_attempts=12, retry_delay=2, interruptible=0, debug=0), load thread runs the REAL dounmount(mp, MNT_FORCE, 0) then immediately re-allocates the freed blocks (same CPU/magazine ⇒ SAME-BLOCK for both, verified) — amp filled with 0x41, node re-gotten from autofs_node_objcache (zeroed):

DF3169: dounmount returned 0 (ticks=15842) -- anp=0xfffff8008d3cd700 amp=0xfffff8008d4be800 ARE FREED NOW
DF3169: reuse blocks: amp 0xfffff8008d4be800 (freed amp was 0xfffff8008d4be800) SAME-BLOCK,
        node 0xfffff8008d3cd700 (freed anp was 0xfffff8008d3cd700) SAME-BLOCK
DF3169: DANGLING REQUEST ar=0xfffff80089942e00 id=1 ar_mount=0xfffff8008d4be800 == FREED amp
DF3169:   ar_from="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"...      <- READ-AFTER-FREE (marker bytes)
DF3169:   ar_path="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"...
DF3169 POLL: freed anp=... an_retries: 1 -> 2 (ticks=16047)  <== WRITE-AFTER-FREE
        ... ten such transitions through ticks=18744 (all > dounmount_ticks=15842)
DF3169 W: trigger returned 60 (ticks=19042, dounmount_ticks=15842)
DF3169: SUMMARY ... dangling=1 last_request_id 0 -> 12

Reads, writes, and the post-free request insertion are all directly observed; ar_from containing the 0x41 marker proves the freed amp block was read after free and after reallocation by unrelated allocation — in production those bytes go to the automount daemon via AUTOFSREQUEST (kernel-heap info leak of reused memory, root-gated reader) and the writes land in whichever kernel object gets the 256 / 4288-byte blocks (silent corruption; an_wildcards is daemon-controlled and callout_reset corrupts callout state).

A first harness revision (with vfs.autofs.debug=2) additionally PANICKED the kernel (panic.txt: Fatal trap 12 … fault va 0x5c … Stopped at autofs_trigger+0x28c) — that specific fault is a harness artifact (kthread curproc == NULL in the AUTOFS_WARN of autofs.c:386-388; real triggers run in process context) but it incidentally proves the predicted interleaving: the drain loop ENXIO-completed W's request mid-dounmount, exactly as traced. The final harness (debug=0, macro body skipped) proceeds past it.

Variant B (fake_daemon.c + variantB.sh): real userspace automountd stand-in (58 requests handled, mkdir storms) × 40 mount/umount -f cycles — no panic in 40 cycles (run_variantB.log); the µs-wide nmkdir window is probabilistic. The corruption variant is the one that fires reliably (KLD run above).

Escalation assessment (primary objective)

The free side of the race requires MNT_FORCE unmount ⇒ privilege gate PR:H. Within that gate the primitive is strong: repeated 4-byte increment at a controlled-ish freed offset, two daemon-influenced bool writes, callout-state corruption, and a request structure whose five MAXPATHLEN string fields are filled from post-free heap — but the attacker who can run umount -f is already root, so there is no unpriv→uid0 chain here. Honest ceiling: kernel heap corruption / panic via a realistic root+user race (e.g. cron automount -u).

Fix validation (see fix_run.log / fix_build.log)

Two fix revisions were built and tested:

• v1 (REJECTED — deadlocks): wrapping the vop bodies in vfs_busy()/vfs_unbusy(). Empirically the patched kernel HUNG inside dounmount() (harness never returned; guest wedged, serial console stopped after "calling dounmount(MNT_FORCE)"). Root cause (code analysis): VOP_NRESOLVE runs with the namecache entry locked (cache_resolve(), sys/kern/vfs_cache.c:4285), so the vop taking mnt_lock-shared creates the order ncp-lock → mnt_lock while dounmount() does mnt_lock(EX) → cache_inval() ncp-locks (sys/kern/vfs_syscalls.c:842→:897) — an AB-BA cycle; a second cycle exists via nlookup()'s inner vfs_busy() which tsleeps until dounmount finishes. See FIX_NOTES.md.
• v2 (SHIPPED, fix.diff — VALIDATED): struct autofs_mount gains am_dying/am_triggers; the three racing vops hold a trigger reference for their whole body (ENXIO fast-fail when dying or force-reclaimed); autofs_unmount() marks the mount dying before the request-drain loop and waits — bounded, aborting with EBUSY — for in-flight triggers before freeing anything. No mnt_lock is taken inside the vops, so neither cycle can form.

Identical harness on the v2-patched kernel (fix_run.log):

DF3169: churn trigger observed (ticks=9476); calling dounmount(MNT_FORCE)
DF3169: dounmount returned 0 (ticks=12571, dounmount_ticks=12571)   <- waited ~31 s for the drain
DF3169: SUMMARY dounmount_ticks=12571 dangling=0 write_after_free_transitions=0
DF3169: CLEAN - no post-free access (fix active?)

vs baseline run.log (same harness, stock kernel): dounmount returned in ~1.3 s with dangling=1, 6 write-after-free transitions, UAF CONFIRMED. Production-shaped stress on the patched kernel (fix_variantB.log): 6/6 mount→trigger+mkdir-storm→umount -f cycles clean (the fake daemon handled requests id≥152 through the fixed path), no panic, no hang; functional sanity (sanity.sh): normal trigger-timeout behavior and plain umount preserved.

Residual (documented, sub-microsecond): the ventry pair (v_flag & VRECLAIMED check / v_mount->mnt_data read) can still race a concurrent force-vgone of the same still-referenced vnode; fully closing it needs a mount-pinning primitive legal with the ncp lock held (vfs-core work for upstream). All seconds-wide windows demonstrated by this finding are closed.

Negative results / notes

• panic.txt documents the artifact panic; not counted as product impact.
• Variant B: 40 cycles no panic — nmkdir window µs-wide.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

struct autofs_mount refcount fix (DF-3169 fix v2)

fix v1 (vfs_busy/vfs_unbusy wrapped around the vop bodies) DEADLOCKS:

• VOP_NRESOLVE is called by cache_resolve() with the namecache entry (ncp) locked exclusively (sys/kern/vfs_cache.c:4285), so taking mnt_lock shared inside the vop creates the order

  trigger thread:   ncp-lock  ->  mnt_lock(shared)
  dounmount:        mnt_lock(EX) -> cache_inval() ncp-locks
                    (sys/kern/vfs_syscalls.c:842 then :897)
  

  a classic AB-BA cycle. It also cycles a second way: the post-trigger nlookup_fs_root() crosses back into the autofs mount, and nlookup's inner vfs_busy() tsleeps until dounmount finishes while dounmount waits for the vop to drop its busy reference. Empirically confirmed: the patched kernel hung inside dounmount() (see VERDICT.md).

fix v2 keeps mnt_lock OUT of the vops entirely:

• struct autofs_mount gains bool am_dying and int am_triggers.
• autofs_nresolve/autofs_readdir/autofs_nmkdir acquire a trigger reference at entry (fail ENXIO if the mount is dying or the vnode was already force-reclaimed) and release it at exit.
• autofs_unmount() marks the mount dying FIRST (new triggers refuse), keeps the existing request-drain loop, then waits for am_triggers to hit zero -- with a bounded wait that ABORTS the unmount with EBUSY (so a trigger thread sleeping in an inner vfs_busy() during dounmount can never deadlock the unmount; the admin retries).
• Only then are the nodes and the autofs_mount freed.

Residual (documented, sub-microsecond): the ventry pair (v_flag&VRECLAIMED check / v_mount->mnt_data read) races a concurrent force-vgone of the still-referenced vnode. Fully closing it needs a mount-pinning primitive that is legal with the ncp lock held (vfs core work); the seconds-wide UAF windows demonstrated by DF-3169 are gone.