DF-2620 — hammer2: crafted on-disk pfs_nmasters drives OOB thread-array access and UAF at unmount (hammer2_xop_helper_cleanup) =======================================================================

CLAIM VERIFIED — both directions reproduced as kernel page-fault panics on the stock INVARIANTS kernel; fix validated by rebuild (panics gone).

What was verified

Root cause confirmed in source: * sys/vfs/hammer2/hammer2_vfsops.c:527-528 — pmp->pfs_nmasters is taken verbatim from the on-disk PFS inode field ripdata->meta.pfs_nmasters (uint8_t, sys/vfs/hammer2/hammer2_disk.h:972) with no clamp to HAMMER2_MAXCLUSTER (=8). * sys/vfs/hammer2/hammer2_admin.c:461-467 — hammer2_xop_helper_cleanup() bounds its teardown loop by pmp->pfs_nmasters while indexing pmp->xop_groups[j].thrs[i], and each group is only thrs[HAMMER2_MAXCLUSTER] wide (sys/vfs/hammer2/hammer2.h:1063-1067).

Reproduction A (up-direction, nmasters=0xFF > 8): * forge: testvol PFS inode meta.pfs_nmasters 0x00 -> 0xFF (forge_df2620.py variant A; CHECK_NONE brefs; volhdr CRCs recomputed). * mount -o ro /dev/vn0@testvol /mnt/h2 -> succeeds * reboot (shutdown -r) -> boot() -> vfs_unmountall -> VFS_UNMOUNT -> hammer2_xop_helper_cleanup walks thrs[0..254] over a 288-slot (18432-byte) allocation: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xfffff801192f5020 supervisor read data, page not present current process = 1 Stopped at hammer2_xop_helper_cleanup+0x5a: cmpq $0,0x20(%rax,%r15,1) The faulting instruction IS the OOB thrs[i].td check (offsetof(hammer2_thread, td) == 0x20). panic.txt, boot_after_A.log.

Reproduction C (down-direction, nmasters=1 < nchains=2): * forge: C.img = clone of base with testvol pfs_type MASTER->SLAVE (same pfs_clid). Two devices carrying one pfs_clid merge into a single pmp: mount vn0@testvol (MASTER, clindex 0), then attempt mount vn1@testvol (SLAVE) — hammer2_update_pmps() merges the SLAVE chain at clindex 1 (creating its 36 xop threads), the mount then fails EBUSY ("PFS already mounted!") but the merged chain stays (the failure path's hammer2_unmount_helper() is a no-op because mount_count != 0). Visible masters = 1 -> pmp->pfs_nmasters = 1 (count bump, vfsops.c:536-542) while nchains = 2. * umount -f /mnt/h2 -> cleanup deletes only the thrs[0] column (i < 1) and immediately kfrees xop_groups with the SLAVE column's 36 live kernel threads still inside; the per-chain cleanup in hammer2_pfsdealloc()/hammer2_pfsfree() (vfsops.c:655-661, 683-690) is skipped because pmp->xop_groups is already NULL. * Result (stock kernel): Fatal trap 12 ... (during umount -f) Stopped at hammer2_primary_xops_thread+0x2d9: lock xaddl %edx,0x81558(%rsi) an xop worker faulting on torn-down state. panic_C.txt, boot_after_C.log, C_run.log.

Also observed (same unclamped ingestion, side effect): * With pfs_nmasters >= 2 on a single-chain mount the mount succeeds but the FIRST access to the mount root hangs forever: nquorum = nmasters/2+1 > 1 chain can ever satisfy -> hammer2_vfs_root() loops at vfsops.c:1966-2008 (wchan "h2root"), pinning the namecache lock of the mountpoint (later path lookups pile up in D-state on "ncplk"). Any umount by path also crosses into VFS_ROOT and hangs the same way. Not fixed by fix.diff (separate robustness bug, noted in the fix header).

Memory-corruption primitive characterization

Up-direction: the walk reads .td at a fixed 64-byte stride across ~15.7KB past an 18KB kmalloc(M_HAMMER2) block; for every OOB slot with non-NULL garbage .td the kernel performs, inside hammer2_thr_delete() (admin.c:254-269): (a) an atomic bit-set write (OR 0x10) into an arbitrary adjacent 32-bit heap word via hammer2_thr_signal() (admin.c:77-97), (b) wakeup() on a garbage address, (c) a NULL write through thr->pmp, (d) kfree(thr->scratch) of a wild 64-bit value, and (e) KKASSERT(TAILQ_EMPTY(&thr->xopq)) over OOB memory. In the observed run the walk page-faulted at the first unmapped page before reaching a populated garbage slot; the intermediate hang observed in an earlier run (umount stuck in "h2twait" forever) is exactly case (a)+(b) on a garbage slot whose flags never gain STOPPED. Down-direction: deterministic UAF — 36 live kernel threads keep polling (30s tsleep loop) and executing from an 18KB block after kfree(), with xopq TAILQ manipulation and xop dispatch through freed memory once the block is reused; stock kernel panics inside the worker (reproduced). A uid=0 chain was not developed: the OOB write offset is stride-locked past a specific allocation and the UAF victim objects are kernel thread structs whose reuse would have to be groomed against a 36-thread poller; both directions reliably panic/DoS instead.

Fix validation (build + boot + re-run, same guest)

Baseline (stock #0, INVARIANTS): trigger A -> Fatal trap 12 in hammer2_xop_helper_cleanup+0x5a (panic.txt); trigger C -> Fatal trap 12 in hammer2_primary_xops_thread+0x2d9 (panic_C.txt). Patched (#1, fix.diff applied to guest /usr/src, nativekernel): * trigger A: mount + shutdown -r -> CLEAN reboot, zero Fatal traps (fixA_run.log; boot.log shows normal syncing disks / Uptime / Rebooting). * trigger C: mount + merge + umount -f -> NO panic, NO corruption, cleanup deletes both columns; the umount ultimately blocks in D-state ("h2twait") waiting for a second-chain worker teardown (fixC_run.log). IMPORTANT: that residual hang is PRE-EXISTING and independent of this finding — a plain unmodified clone pair (MASTER+MASTER, nmasters == nchains == 2, no forged bytes, clamp inert, cleanup sequence byte-identical to stock) hangs umount -f identically (verified, wchan "h2twait"). It is a separate multi-chain teardown defect worth its own finding; on the STOCK kernel trigger C did not hang — it corrupted memory and panicked instead.

Files

forge_df2620.py image forger (variants A/B/C) — host python3 df2620_dump.py.in walking dumper used to verify the on-disk layout trigger_A.sh up-direction trigger (mount + reboot) trigger_B.sh lone-SLAVE trigger (wedges VFS_ROOT; kept for record) trigger_C.sh down-direction trigger (2-device merge + umount -f) fixA.sh/fixC.sh fix-validation triggers fix.diff the verified fix (clamp + cluster-width bound) panic.txt stock kernel, variant A trap + ddb stop panic_C.txt stock kernel, variant C trap + ddb stop boot_after_A.log / boot_after_C.log full serial console logs C_run.log variant C run log (stock; ends at the fatal umount) fixA_run.log / fixC_run.log fix-validation run logs (patched) fix_build.log full untrimmed nativekernel build log (37793 lines) env.txt kernels, sysctls, md5s

Exact reproduction (from this repo root)

forge (needs the base image: newfs_hammer2 -L testvol on a 64M file)

python3 findings/poc/DF-2620/forge_df2620.py base.img A A.img python3 findings/poc/DF-2620/forge_df2620.py base.img C C.img

in guest (root): variant A

vnconfig -c vn0 A.img && mkdir -p /mnt/h2 mount -t hammer2 -o ro /dev/vn0@testvol /mnt/h2 shutdown -r now # -> Fatal trap 12, hammer2_xop_helper_cleanup+0x5a

in guest (root): variant C

vnconfig -c vn0 base.img; vnconfig -c vn1 C.img mount -t hammer2 -o ro /dev/vn0@testvol /mnt/h2 mount -t hammer2 -o ro /dev/vn1@testvol /mnt/h2b # EBUSY (chain merged) umount -f /mnt/h2 # -> Fatal trap 12, hammer2_primary_xops_thread+0x2d9