DF-2621 — hammer2_xop_helper_create() unconditional re-kmalloc: leaked thread arrays + orphaned kernel threads (pmp UAF blocked at runtime) ===================================================================== VERDICT: REPRODUCED (leak + permanent orphan kernel threads, deterministic, root-gated on this build). The claimed use-after-free of the kfree'd pmp is PROVEN IN CODE and fully instrumented, but could not be observed live on this guest because an independent, pre-existing multi-chain teardown defect wedges every 2-chain pmp teardown in `hammer2_pfsfree_scan()`'s freeze phase before `hammer2_pfsfree()` ever runs. Fix validated by guest rebuild: the double-create, the leak and the orphan threads are gone. 1. Root cause confirmed in source --------------------------------- * sys/vfs/hammer2/hammer2_admin.c:425-448 — `hammer2_xop_helper_create()` executes `pmp->xop_groups = kmalloc(...)` (admin.c:434-436) with no NULL check, overwriting any previous array. The inner per-slot guard (`if (pmp->xop_groups[j].thrs[i].td) continue;`, admin.c:439) is defeated because the fresh array is M_ZERO'd. * Call sites that collide on one pmp: - sys/vfs/hammer2/hammer2_vfsops.c:588-589 — `hammer2_pfsalloc()` merging a same-`pfs_clid` chain into a pmp: `if (pmp->mp || iroot->cluster.nchains >= 2) hammer2_xop_helper_create(pmp);` - sys/vfs/hammer2/hammer2_vfsops.c:1708 — `hammer2_mount_helper()` (unconditional), - sys/vfs/hammer2/hammer2_admin.c:491-492 — lazy re-create from `hammer2_xop_start_except()` when `has_xop_threads == 0`. * Merge reachability: `hammer2_pfsalloc()` matches pmps by `pfs_clid` (vfsops.c:399-410), appends the chain (nchains 1→2) and bumps the new device's mount_count (vfsops.c:505-506). The *mount* then fails EBUSY at vfsops.c:1454-1459 — but the merge (and the helper_create calls) already happened and persist. * Teardown only ever walks the CURRENT `pmp->xop_groups`: hammer2_xop_helper_cleanup (admin.c:461-469), hammer2_pfsdealloc (vfsops.c:657-662), hammer2_pfsfree_scan (vfsops.c:776-819), hammer2_pfsfree (vfsops.c:693-699). The previous array's threads are never signalled HAMMER2_THREAD_STOP; nothing in the kernel retains a pointer to them. Each orphan holds `thr->scratch` = kmalloc(MAXPHYS = 128 KB) (admin.c:233-234), freed only by hammer2_thr_delete (admin.c:261-263) — never called for orphans. 2. What was reproduced, on which kernel --------------------------------------- All runs: QEMU/KVM guest, DragonFly 6.5-DEVELOPMENT x86_64, stock INVARIANTS kernel #0 (and rebuilt #1 kernels), 6 vCPUs, hammer2 root fs. `hammer2_xop_nthreads` = 36 on this box (vfsops.c:258-267 ⇒ 6 cpus); xop_groups array = 36 × 8 × 64 B = 18432 B. Trigger (NO forged bytes needed): `truncate -s 64M base.img; newfs_hammer2 -L testvol base.img; dd` a byte-identical clone (identical pfs_clid by construction); `vnconfig -c vn0/vn1`; mount vn0@testvol, then attempt mount vn1@testvol. STOCK kernel #0: * mount1 ⇒ 36 threads ("h2xop-testvol.00..35"). mount2 ⇒ EBUSY, census 108 = 36 orphans + 72 new (create at nchains=2 makes 2×36). Console shows both mounts binding the SAME pmp (run1: 0xfffff80118d80000, run2: 0xfffff80118be0000, run3: 0xfffff80118c40000). Deterministic across all three vulnerable-kernel runs (run.log, run2.log, run3.log). * Amplification (run3.log): 4 devices ⇒ census 0 → 36 → 108 → 216 → 360 (sequential creates at nchains=1,2,3,4 add 36/72/108/144 threads; 216 of them orphaned forever, all parked in "h2idle" — the 30 s poll sleep). `vmstat -m` HAMMER2-mount malloc zone: 13.0M → 130M (+117 MB) from eight mount commands, with NO unmount and NO filesystem activity. Leak is unbounded in device count (cluster-full merges still call helper_create — it sits outside the nchains guard at vfsops.c:583-589). * Orphans are permanent: they appear in every post-teardown census for the life of the boot; no signal path exists that could stop them. INSTRUMENTED kernel #1 (instrument.diff, in-guest only): decisive console trace (instrumented_console.log): DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0 nchains=1 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80118b4e000 DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0xfffff80118b4e000 nchains=2 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80119a70000 (OLD LEAKED) [... mount2 → "PFS already mounted!" ...] DF2621: helper_cleanup enter pmp=... groups=0xfffff80119a70000 nmasters=2 DF2621: helper_cleanup col 0 deleted DF2621: helper_cleanup col 1 deleted DF2621: helper_cleanup freeing groups=0xfffff80119a70000 DF2621: pfsfree_scan hmp=0xfffff801184a0000 which=0 enter DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0 nchains=2 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80118e00000 (OLD LEAKED) DF2621: pfsfree_scan freeze begin pmp=0xfffff80118c80000 groups=0xfffff80118e00000 [umount wedges here forever] This proves: (a) double-create on the same pmp (old_groups non-NULL, overwritten); (b) the CURRENT array is torn down correctly; (c) a THIRD array is lazily re-created mid-teardown by xop_start (admin.c:491) during pfsfree_scan's own sync — the source of the 72 "frozen" threads seen in the stock wedges; (d) the wedge is in pfsfree_scan's freeze loop (vfsops.c:783-793), BEFORE any pmp kfree. 3. The UAF claim — honest assessment ------------------------------------ Code path (verified line-by-line): an orphaned worker's poll loop (admin.c:1158) calls `hammer2_xop_next(thr)` UNCONDITIONALLY each iteration — the HAMMER2_THREAD_XOPQ check at admin.c:1211 only clears the flag and falls through — and hammer2_xop_next executes `hammer2_spin_ex(&pmp->xop_spin)` (admin.c:1074-1079) before scanning `thr->xopq` (which lives in the leaked-but-still-allocated array, so that part is not a fault). After `tsleep(..., "h2idle", hz*30)` (admin.c:1244) each orphan therefore performs a spinlock acquire+release WRITE against `thr->pmp` — every 30 seconds, forever. If `hammer2_pfsfree()` kfrees the pmp (vfsops.c:722) while orphans exist, those are writes into freed kernel heap at a fixed offset inside a several-KB M_HAMMER2 block, plus a TAILQ scan; a reused word that resembles a held spinlock would additionally make an orphan spin forever against a real victim object. Runtime: NOT observed. `hammer2_pfsfree()` is unreachable for any 2-chain pmp on this kernel because every teardown (plain umount, umount -f, and shutdown) wedges earlier in `hammer2_pfsfree_scan()`'s freeze phase. This wedge is a SEPARATE, pre-existing defect (DF-2620 already observed the identical h2twait wedge with an unrelated fix applied; my breadcrumbs localize it to the freeze loop after the mid-teardown lazy re-create). The instrumented freed-pmp ring + detector I added (`df2621_check_freed()` in hammer2_xop_next) never fired for exactly this reason — the pmp never entered the ring. The double-create is a necessary but not sufficient condition for the UAF on this kernel generation; on a kernel where multi-chain teardown completes, the orphan writes to the freed pmp follow mechanically from admin.c:1074-1079 + vfsops.c:722. Also NOT observed: any panic attributable to DF-2621. (The stock umount wedge leaves the system otherwise usable, but hammer2 mntlk stays held by the wedged unmount thread, blocking all later hammer2 mounts/unmounts, and shutdown hangs — the guest had to be force-killed on every reset.) 4. Threat model check (run4.log) --------------------------------- The finding's unprivileged precondition did NOT hold on this guest: with `vfs.usermount=1`, /dev/vn0 and /dev/vn1 chown'd (and chmod 660) to the unprivileged user, and a user-owned mountpoint, `mount -t hammer2` still returns EPERM for the user. The trigger therefore requires mount privilege on this build (root, or a privileged automount path mounting attacker-supplied media such as a dd-duplicated device). The double-create also fires from ordinary privileged cluster administration (mounting a second device carrying the same clid), making the leak a correctness bug even absent an attacker. 5. Fix validation (guest rebuild) --------------------------------- fix.diff guards the allocation (`if (pmp->xop_groups == NULL)`) and keeps the existing create-missing-threads loop. Applied to a pristine guest /usr/src, `make nativekernel && make installkernel` completed BUILD_OK (fix_build.log), rebooted into kernel #1. Baseline (stock #0, same trigger): T2 = 108 threads (36 orphans + 72), post-wedge census 36 h2idle orphans + 72 frozen. Patched (#1, same trigger): T2 = 72 threads (fix_run.log: threads=72, idle=72, frozen=0) — the merge added ONLY the missing clindex-1 column to the EXISTING array; during the (still present, pre-existing) teardown wedge the census is 0 h2idle orphans + 72 frozen. No orphan threads exist at any point. The pre-existing teardown wedge itself is unaffected by the guard, as expected — it is a separate defect (lazy re-create mid-teardown + freeze-phase hang) and out of scope for this finding. fix verdict: fixed — the double-create, the array/thread/scratch leak and the orphan threads are eliminated; the unrelated teardown wedge remains (recommend a separate finding; see note in fix.diff header). 6. Primitive characterization (if the UAF were reachable) -------------------------------------------------------- Write primitive per orphan per 30 s: spin_lock/spin_unlock pair on the 32-bit word at offsetof(hammer2_pfs, xop_spin) (hammer2.h:1244) of a freed, reusable M_HAMMER2 block of several KB (sizeof(hammer2_pfs) with 8 embedded sync threads + inumhash). Lock value transitions 0→1→0 (plus spin-loop reads while contended). Secondary: TAILQ_FOREACH over thr->xopq (leaked array — allocated, not freed). A uid=0 chain was not developed: the write is a transient bit pattern on a fixed offset with a 30 s cadence and no length control — corruption-class, not a clean arbitrary-write — and on this guest the pmp never reaches kfree (hard blocker, documented above). 7. Files -------- trigger.sh stock v1: mount/merge/EBUSY/umount (plain) — first wedge trigger2.sh stock v2: umount -f variant + idle/frozen census trigger3.sh stock v3: 4-device amplification (leak quantification) trigger4.sh usermount/unprivileged attempt (EPERM — threat model check) instrument.diff kprintf breadcrumbs + freed-pmp ring/UAF detector (guest only) fix.diff the validated one-line-guard fix (+ comment) run.log / run2.log(=fix_run.log counterpart) / run3.log / run4.log run2_instrumented.log full trigger2 run on instrumented kernel instrumented_console.log decisive breadcrumb trace fix_run.log trigger2 on fix kernel (T2=72, orphans=0) fix_console.log fix-kernel mount console (same pmp both mounts) fix_build.log full untrimmed fix-kernel build env.txt guest environment