DF-3250 / verdict.json
{ "finding_id": "DF-3250", "status": "reproduced", "reproduced": 1, "impact": "panic", "confidence": "certain", "verdict": "Unprivileged-class kernel heap double-free in ext2_alloc_vnodes EBUSY-collision handler, triggered by racing concurrent ext2 inode instantiations: vp->v_data is published before ext2_ihashins (ext2_vfsops.c:1297 vs :1313), so the handlers vx_put() synchronously reclaims through ext2_reclaim which frees the inode (ext2_inode.c:569), and the handlers own free(ip) at :1319 frees it again - verified as panic: memory chunk ... is already free! (_kfree <- ext2_alloc_vnode <- ext2_vget <- ext2_fhtovp) after 32,757 raced cold creations on the stock INVARIANTS guest. Sibling flaw in the same hand-rolled ext2fs_inode_hash_lock: lost wakeup wedges syscalls permanently in D-state (observed live). Fix validated A/B: with v_data assigned only after a successful insert (fix.diff), the same EBUSY branch fired again (ino=23506) and the guest survived 1.17M+ further creations.", "exploit_chain": "two CPU-pinned processes barrier-synchronize fhopen()/open() of the same cold ext2 inode (hardlinked names in two directories avoid parent-dir lock serialization; kern.maxvnodes pressure forces VOP_RECLAIM so inodes cycle cold) -> both ext2_vget creators miss ext2_ihashget -> ext2fs_inode_hash_lock does not cover the miss and is non-atomic -> loser passes the check after winners unlock -> ext2_ihashins returns EBUSY -> vx_put(VBAD, last ref) sets VREF_FINALIZE -> synchronous vrecycle -> ext2_reclaim frees ip #1 (+ unbalanced vrele(um_devvp)) -> handler free(ip) #2 -> slab free-chunk panic. On non-checking builds the second free is exploitable heap corruption (chunk double-handed to the magazine); the unbalanced device-vnode vrele enables premature devvp death under repeat.", "evidence": [ "panic.txt: console capture - DF3250[ihash]: EBUSY! dev=0xfffff8004f2bb780 ino=30751 (ins=32757 ebusy=1), ext2_alloc_vnode: ihashins collision, retrying inode 30751, panic: memory chunk 0xfffff80119113620 is already free!, stack _kfree <- ext2_alloc_vnode <- ext2_vget <- ext2_fhtovp", "fix_dmesg.txt / fix_counters.txt: post-fix run, same EBUSY event (ino=23506 ... ebusy=1) with guest alive; debug.df3250_alloc=1176322, debug.df3250_slept=4569, debug.df3250_ebusy=1, no panic", "fh_race.c: fhopen-based race driver (pinned pair, per-round phase sweep 0..250us, barrier stall detection for the lost-wakeup wedge)", "race_e2.c: unprivileged open()-path driver via two hardlinked names in two directories", "instrument.diff: counter/probe patch to ext2fs.ko used for both baseline and fix runs", "VERDICT.md: full A/B narrative and path:line trace" ], "kernel_refs": [ "sys/vfs/ext2fs/ext2_vfsops.c:1297", "sys/vfs/ext2fs/ext2_vfsops.c:1313", "sys/vfs/ext2fs/ext2_vfsops.c:1318", "sys/vfs/ext2fs/ext2_vfsops.c:1319", "sys/vfs/ext2fs/ext2_vfsops.c:1275", "sys/vfs/ext2fs/ext2_vfsops.c:1325", "sys/vfs/ext2fs/ext2_inode.c:562", "sys/vfs/ext2fs/ext2_inode.c:569", "sys/vfs/ext2fs/inode.h:174", "sys/kern/vfs_lock.c:763", "sys/kern/vfs_subr.c:1498", "sys/vfs/ufs/ffs_vfsops.c:1120" ], "poc_changes": "Original concept (open() race) rewrote three times: (1) same-directory opens serialize on the parent dir vnode lock - replaced by hardlinked names in two directories; (2) cached-but-inactive vnodes stay in the ext2 ihash and are revived by vget, so cold supply requires forced VOP_RECLAIM (kern.maxvnodes pressure and/or MNT_RELOAD purge via direct mount(2), reload.c; mount(8) rejects -o reload for ext2fs); (3) final driver uses fhopen() (root-gated convenience; VFS_FHTOVP -> ext2_vget directly, no namei jitter), CPU pinning via usched_set (root-only), per-round TSC phase sweep, and a supervisor that respawns the pair when the kernel-side lost-wakeup flaw (same broken lock) wedges a thread permanently. Kernel instrumentation (instrument.diff): counters + printf on the EBUSY branch only, no behavior change; used identically for baseline and fix runs.", "attempts": 7, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 5400, "guest_dirty": 0, "build_cmd": "host: dd+mkfs.ext2 image, scp; guest: tar xzf mod.tgz; cd e2mod && touch opt_suiddir.h && make; cc -O2 -o fh_race fh_race.c; cc -O2 -o race_e2 race_e2.c; kldload e2mod/ext2fs.ko; vnconfig -c vn0 e2.img; mount -t ext2fs /dev/vn0 /mnt/e2; populate 6000 hardlink pairs dA/dB; mount -u -o rdonly; sysctl kern.maxvnodes=1200", "run_cmd": "sh /root/supervise.sh & ; sh /root/supervise.sh & ; watch sysctl -a | grep df3250 and console (vm.sh log) - baseline expects panic within ~30-120s; fixed module expects EBUSY message with guest alive", "code_hash": "87a9dc04408dbeb70d1728f5628042de1f0945875d249d4956174b5e43a1ab32", "notes": "Trigger rate ~1 EBUSY per 33k-830k raced cold creations (Poisson; luck-dependent). The instrumented module is behavior-identical (counters/printf only) - the panic comes from unmodified stock code paths. The unbalanced vrele(um_devvp) rides the same reclaim and is fixed by the same ordering change. Two guest resets were consumed by earlier harness wedges (unmount-vs-racers; pkill of a thread inside the critical section leaks hash_lock=1 - a harness artifact, documented in VERDICT.md). Follow-up recommendations beyond fix.diff: make ext2fs_inode_hash_lock set/clear atomic (closes the permanent lost-wakeup D-state hang) or fold creator serialization into ext2_ihash_token.", "recommended_fix": "Assign vp->v_data only after ext2_ihashins() succeeds (mirror ffs_vfsops.c:1120-1129 ordering) so the EBUSY handlers vx_put-triggered synchronous reclaim sees v_data == NULL and neither walks nor frees the handler-owned inode.", "fix_status": "fixed", "fix_kernel_uname": "DragonFly dfbsd 6.5-DEVELOPMENT #0 (stock kernel; ext2fs.ko rebuilt from fixed sources, module-level)", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "A/B validated on the same guest: baseline module panicked deterministically on the first EBUSY (ins=32757); fixed module (fix.diff, one hunk moving vp->v_data assignment after ext2_ihashins) hit the same EBUSY branch (ino=23506, ins=829996, ebusy=1 printed to console) and survived - guest alive after 1.17M+ further creations with 4569 lock-contention pairings.", "fix_evidence": [ "fix.diff", "fix_counters.txt", "fix_dmesg.txt", "VERDICT.md run table" ] } |