DF-2621 — hammer2_xop_helper_create() unconditional re-kmalloc leaks thread

arrays + orphan kernel threads (verified)

Claim under test (from findings/DF-2621-hammer2-xop-helper-create-double-alloc-uaf.md): hammer2_xop_helper_create() (sys/vfs/hammer2/hammer2_admin.c:425-448) assigns pmp->xop_groups = kmalloc(...) unconditionally. When called twice on the same pmp — which happens whenever a second same-pfs_clid device merges into a pmp that already has xop threads (hammer2_pfsalloc, vfsops.c:588-589, then hammer2_mount_helper, vfsops.c:1708) — the old array pointer is lost: the ~18 KB array, its 36 kernel threads and their 36 x 128 KB scratch buffers are leaked, and the threads are orphaned (all teardown paths walk only the CURRENT pmp->xop_groups). The finding further claims the orphans use-after-free the pmp after hammer2_pfsfree() kfrees it (30 s poll loop touches pmp->xop_spin, admin.c:1244/1079).

Verified results (full story: VERDICT.md)

1. Double-create on the same pmp — PROVEN (stock console: both mounts bind the same pmp; instrumented kernel: old_groups non-NULL, overwritten).
2. Deterministic leak + orphan threads — PROVEN on stock: thread census 0 -> 36 -> 108 per single clone-merge; with 4 devices 0 -> 36 -> 108 -> 216 -> 360 (creates of 36/72/108/144 threads; 216 orphaned), HAMMER2-mount malloc zone 13.0M -> 130M (+117 MB) in eight mount commands, no unmount involved, guest otherwise idle. Orphan threads survive every teardown path.
3. UAF-write component — code-proven, runtime-blocked: orphan threads unconditionally execute hammer2_spin_ex(&pmp->xop_spin) every 30 s poll (admin.c:1218 falls through into 1074-1079); nothing ever signals them STOP. hammer2_pfsfree() would kfree the pmp while they poll. On this guest the pmp never reaches kfree: every 2-chain teardown wedges earlier in hammer2_pfsfree_scan()'s freeze phase (a separate, pre-existing multi-chain teardown defect, already documented with DF-2620), so the freed-pmp write could not be observed live. See VERDICT.md.
4. Fix validated by rebuild: guarding the kmalloc with if (pmp->xop_groups == NULL) removes the double-create (T2 108 -> 72), leaves zero orphan threads, and stops the leak growth. fix.diff.

Reproduce

Images need NO forged bytes — a dd clone shares the pfs_clid by construction.

# in-guest as root (stock INVARIANTS kernel #0):
sh trigger2.sh     # mount vn0@testvol; mount vn1@testvol (dd clone) -> EBUSY;
                   # census shows 36+72 threads; umount -f wedges (separate bug)
sh trigger3.sh     # amplification: 4 devices, census to 360 threads,
                   # vmstat -m HAMMER2-mount 13M -> 130M

Success criteria: * MOUNT2_RC=1 with console hammer2_mount: ... pmp=<same addr> + PFS already mounted! * thread census T2=108 (36 orphans + 72) vs fixed kernel T2=72 * vmstat -m | grep HAMMER2-mount grows monotonically per clone-merge

Instrumented proof build

instrument.diff adds kprintf breadcrumbs (helper_create old/new pointers, cleanup progress, pfsfree_scan phases, pfsfree kfree) and a freed-pmp ring checked in hammer2_xop_next() that would report an orphan spinning on a freed pmp. Applied to the guest /usr/src only, never to the audit tree. Decisive output in instrumented_console.log:

DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0 nchains=1
DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80118b4e000
DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0xfffff80118b4e000 nchains=2
DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80119a70000 (OLD LEAKED)

DF-2621 — hammer2_xop_helper_create() unconditional re-kmalloc: leaked thread arrays + orphaned kernel threads (pmp UAF blocked at runtime) =====================================================================

VERDICT: REPRODUCED (leak + permanent orphan kernel threads, deterministic, root-gated on this build). The claimed use-after-free of the kfree'd pmp is PROVEN IN CODE and fully instrumented, but could not be observed live on this guest because an independent, pre-existing multi-chain teardown defect wedges every 2-chain pmp teardown in hammer2_pfsfree_scan()'s freeze phase before hammer2_pfsfree() ever runs. Fix validated by guest rebuild: the double-create, the leak and the orphan threads are gone.

1. Root cause confirmed in source

• sys/vfs/hammer2/hammer2_admin.c:425-448 — hammer2_xop_helper_create() executes pmp->xop_groups = kmalloc(...) (admin.c:434-436) with no NULL check, overwriting any previous array. The inner per-slot guard (if (pmp->xop_groups[j].thrs[i].td) continue;, admin.c:439) is defeated because the fresh array is M_ZERO'd.
• Call sites that collide on one pmp:
• sys/vfs/hammer2/hammer2_vfsops.c:588-589 — hammer2_pfsalloc() merging a same-pfs_clid chain into a pmp: if (pmp->mp || iroot->cluster.nchains >= 2) hammer2_xop_helper_create(pmp);
• sys/vfs/hammer2/hammer2_vfsops.c:1708 — hammer2_mount_helper() (unconditional),
• sys/vfs/hammer2/hammer2_admin.c:491-492 — lazy re-create from hammer2_xop_start_except() when has_xop_threads == 0.
• Merge reachability: hammer2_pfsalloc() matches pmps by pfs_clid (vfsops.c:399-410), appends the chain (nchains 1→2) and bumps the new device's mount_count (vfsops.c:505-506). The mount then fails EBUSY at vfsops.c:1454-1459 — but the merge (and the helper_create calls) already happened and persist.
• Teardown only ever walks the CURRENT pmp->xop_groups: hammer2_xop_helper_cleanup (admin.c:461-469), hammer2_pfsdealloc (vfsops.c:657-662), hammer2_pfsfree_scan (vfsops.c:776-819), hammer2_pfsfree (vfsops.c:693-699). The previous array's threads are never signalled HAMMER2_THREAD_STOP; nothing in the kernel retains a pointer to them. Each orphan holds thr->scratch = kmalloc(MAXPHYS = 128 KB) (admin.c:233-234), freed only by hammer2_thr_delete (admin.c:261-263) — never called for orphans.

2. What was reproduced, on which kernel

All runs: QEMU/KVM guest, DragonFly 6.5-DEVELOPMENT x86_64, stock INVARIANTS kernel #0 (and rebuilt #1 kernels), 6 vCPUs, hammer2 root fs. hammer2_xop_nthreads = 36 on this box (vfsops.c:258-267 ⇒ 6 cpus); xop_groups array = 36 × 8 × 64 B = 18432 B.

Trigger (NO forged bytes needed): truncate -s 64M base.img; newfs_hammer2 -L testvol base.img; dd a byte-identical clone (identical pfs_clid by construction); vnconfig -c vn0/vn1; mount vn0@testvol, then attempt mount vn1@testvol.

STOCK kernel #0: * mount1 ⇒ 36 threads ("h2xop-testvol.00..35"). mount2 ⇒ EBUSY, census 108 = 36 orphans + 72 new (create at nchains=2 makes 2×36). Console shows both mounts binding the SAME pmp (run1: 0xfffff80118d80000, run2: 0xfffff80118be0000, run3: 0xfffff80118c40000). Deterministic across all three vulnerable-kernel runs (run.log, run2.log, run3.log). * Amplification (run3.log): 4 devices ⇒ census 0 → 36 → 108 → 216 → 360 (sequential creates at nchains=1,2,3,4 add 36/72/108/144 threads; 216 of them orphaned forever, all parked in "h2idle" — the 30 s poll sleep). vmstat -m HAMMER2-mount malloc zone: 13.0M → 130M (+117 MB) from eight mount commands, with NO unmount and NO filesystem activity. Leak is unbounded in device count (cluster-full merges still call helper_create — it sits outside the nchains guard at vfsops.c:583-589). * Orphans are permanent: they appear in every post-teardown census for the life of the boot; no signal path exists that could stop them.

INSTRUMENTED kernel #1 (instrument.diff, in-guest only): decisive console trace (instrumented_console.log): DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0 nchains=1 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80118b4e000 DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0xfffff80118b4e000 nchains=2 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80119a70000 (OLD LEAKED) [... mount2 → "PFS already mounted!" ...] DF2621: helper_cleanup enter pmp=... groups=0xfffff80119a70000 nmasters=2 DF2621: helper_cleanup col 0 deleted DF2621: helper_cleanup col 1 deleted DF2621: helper_cleanup freeing groups=0xfffff80119a70000 DF2621: pfsfree_scan hmp=0xfffff801184a0000 which=0 enter DF2621: helper_create pmp=0xfffff80118c80000 old_groups=0 nchains=2 DF2621: helper_create pmp=0xfffff80118c80000 new_groups=0xfffff80118e00000 (OLD LEAKED) DF2621: pfsfree_scan freeze begin pmp=0xfffff80118c80000 groups=0xfffff80118e00000 [umount wedges here forever] This proves: (a) double-create on the same pmp (old_groups non-NULL, overwritten); (b) the CURRENT array is torn down correctly; (c) a THIRD array is lazily re-created mid-teardown by xop_start (admin.c:491) during pfsfree_scan's own sync — the source of the 72 "frozen" threads seen in the stock wedges; (d) the wedge is in pfsfree_scan's freeze loop (vfsops.c:783-793), BEFORE any pmp kfree.

3. The UAF claim — honest assessment

Code path (verified line-by-line): an orphaned worker's poll loop (admin.c:1158) calls hammer2_xop_next(thr) UNCONDITIONALLY each iteration — the HAMMER2_THREAD_XOPQ check at admin.c:1211 only clears the flag and falls through — and hammer2_xop_next executes hammer2_spin_ex(&pmp->xop_spin) (admin.c:1074-1079) before scanning thr->xopq (which lives in the leaked-but-still-allocated array, so that part is not a fault). After tsleep(..., "h2idle", hz*30) (admin.c:1244) each orphan therefore performs a spinlock acquire+release WRITE against thr->pmp — every 30 seconds, forever. If hammer2_pfsfree() kfrees the pmp (vfsops.c:722) while orphans exist, those are writes into freed kernel heap at a fixed offset inside a several-KB M_HAMMER2 block, plus a TAILQ scan; a reused word that resembles a held spinlock would additionally make an orphan spin forever against a real victim object.

Runtime: NOT observed. hammer2_pfsfree() is unreachable for any 2-chain pmp on this kernel because every teardown (plain umount, umount -f, and shutdown) wedges earlier in hammer2_pfsfree_scan()'s freeze phase. This wedge is a SEPARATE, pre-existing defect (DF-2620 already observed the identical h2twait wedge with an unrelated fix applied; my breadcrumbs localize it to the freeze loop after the mid-teardown lazy re-create). The instrumented freed-pmp ring + detector I added (df2621_check_freed() in hammer2_xop_next) never fired for exactly this reason — the pmp never entered the ring. The double-create is a necessary but not sufficient condition for the UAF on this kernel generation; on a kernel where multi-chain teardown completes, the orphan writes to the freed pmp follow mechanically from admin.c:1074-1079 + vfsops.c:722.

Also NOT observed: any panic attributable to DF-2621. (The stock umount wedge leaves the system otherwise usable, but hammer2 mntlk stays held by the wedged unmount thread, blocking all later hammer2 mounts/unmounts, and shutdown hangs — the guest had to be force-killed on every reset.)

4. Threat model check (run4.log)

The finding's unprivileged precondition did NOT hold on this guest: with vfs.usermount=1, /dev/vn0 and /dev/vn1 chown'd (and chmod 660) to the unprivileged user, and a user-owned mountpoint, mount -t hammer2 still returns EPERM for the user. The trigger therefore requires mount privilege on this build (root, or a privileged automount path mounting attacker-supplied media such as a dd-duplicated device). The double-create also fires from ordinary privileged cluster administration (mounting a second device carrying the same clid), making the leak a correctness bug even absent an attacker.

5. Fix validation (guest rebuild)

fix.diff guards the allocation (if (pmp->xop_groups == NULL)) and keeps the existing create-missing-threads loop. Applied to a pristine guest /usr/src, make nativekernel && make installkernel completed BUILD_OK (fix_build.log), rebooted into kernel #1.

Baseline (stock #0, same trigger): T2 = 108 threads (36 orphans + 72), post-wedge census 36 h2idle orphans + 72 frozen. Patched (#1, same trigger): T2 = 72 threads (fix_run.log: threads=72, idle=72, frozen=0) — the merge added ONLY the missing clindex-1 column to the EXISTING array; during the (still present, pre-existing) teardown wedge the census is 0 h2idle orphans + 72 frozen. No orphan threads exist at any point. The pre-existing teardown wedge itself is unaffected by the guard, as expected — it is a separate defect (lazy re-create mid-teardown + freeze-phase hang) and out of scope for this finding.

fix verdict: fixed — the double-create, the array/thread/scratch leak and the orphan threads are eliminated; the unrelated teardown wedge remains (recommend a separate finding; see note in fix.diff header).

6. Primitive characterization (if the UAF were reachable)

Write primitive per orphan per 30 s: spin_lock/spin_unlock pair on the 32-bit word at offsetof(hammer2_pfs, xop_spin) (hammer2.h:1244) of a freed, reusable M_HAMMER2 block of several KB (sizeof(hammer2_pfs) with 8 embedded sync threads + inumhash). Lock value transitions 0→1→0 (plus spin-loop reads while contended). Secondary: TAILQ_FOREACH over thr->xopq (leaked array — allocated, not freed). A uid=0 chain was not developed: the write is a transient bit pattern on a fixed offset with a 30 s cadence and no length control — corruption-class, not a clean arbitrary-write — and on this guest the pmp never reaches kfree (hard blocker, documented above).

7. Files

trigger.sh stock v1: mount/merge/EBUSY/umount (plain) — first wedge trigger2.sh stock v2: umount -f variant + idle/frozen census trigger3.sh stock v3: 4-device amplification (leak quantification) trigger4.sh usermount/unprivileged attempt (EPERM — threat model check) instrument.diff kprintf breadcrumbs + freed-pmp ring/UAF detector (guest only) fix.diff the validated one-line-guard fix (+ comment) run.log / run2.log(=fix_run.log counterpart) / run3.log / run4.log run2_instrumented.log full trigger2 run on instrumented kernel instrumented_console.log decisive breadcrumb trace fix_run.log trigger2 on fix kernel (T2=72, orphans=0) fix_console.log fix-kernel mount console (same pmp both mounts) fix_build.log full untrimmed fix-kernel build env.txt guest environment