DF-3168 — hammer rebalance count-trust heap overflow

Crafted-image HAMMER1 B-Tree leaf with count > 63 (CRC recomputed by the image author) → hammer rebalance (the HAMMER_IOC_REBALANCE ioctl behind hammer cleanup / hammer rebalance) packs elements using the unvalidated on-disk count, driving avg_elms past the 63-element elms[] array of the kmalloc'd 4KB node copies → out-of-bounds kernel heap write.

Contents

• mkimage.sh — stage 1: build a legitimate 12G HAMMER1 image with a multi-leaf B-Tree (~2500 dirents)
• patch_leaf_rebal.c — stage 2: walk the image B-Tree, select a suitable parent (first child-index at the root) and forge count=128 on its conforming non-first leaf children (slice-0/1 leaves, in-buffer elms[127] scan-read), re-CRC the nodes; verify mode reads the counts back after a run
• trigger.sh — stage 3: mount rw, hammer rebalance, umount, verify, remount + search (panic expected on stock kernel)
• trigger_fix.sh — fix-validation variant (stops after verify)
• console_full.log — FULL serial console of the baseline run
• panic.txt — decisive excerpt (add count 128 ... _kfree_obj panic)
• fixrun.log — full fixed-kernel run
• build.log — guest build of patch_leaf_rebal
• fix.diff — the validated fix (git-apply-able vs sys/)

Build (in guest)

cc -O2 -o /tmp/patch_leaf_rebal patch_leaf_rebal.c

Run (in guest, as root)

sh mkimage.sh
/tmp/patch_leaf_rebal walk   /build/poc.img   # pick targets
/tmp/patch_leaf_rebal patch  /build/poc.img   # forge count=128 + re-CRC
sh trigger.sh                                  # baseline: panic

Expected

Stock INVARIANTS kernel (baseline, console_full.log):

rebalance_node: lockroot ... count 31
rebalance_node: add count 128        (x14 -- unvalidated disk counts)
rebalance_closeout: 8000000021000000: (count update) ... (x31)
panic: assertion "slab->magic == KMALLOC_SLAB_MAGIC && delta != slab->ncount"
       failed in _kfree_obj at kern/kern_kmalloc.c:942
_kfree_obj -> fdrop -> closef -> kern_close -> syscall2

The rebalance pack loop wrote 64-byte granules past the end of the 4096-byte kmalloc'd copy buffers (base_count ran to avg_elms=76 > 63), corrupting adjacent kernel heap / slab metadata; an unrelated close() syscall detected it. On production kernels (no INVARIANTS slab-magic check) the overflow is silent → attacker-influenced heap corruption (see VERDICT.md for the primitive characterization).

Fixed kernel (fixrun.log):

hammer_ioc_rebalance: rebalance: bad node 8000000021004000 type L count 128
Rebalance /mnt failed: Input/output error
VERIFY: 14 out-of-bounds counts found     (only the leaves we forged)

DF-3168 — VERDICT

Status: reproduced (stock INVARIANTS kernel #0 Jul 2 06:02:54) — fix validated (patched kernel #1 Sep 7 13:18:26).

The bug

hammer_ioc_rebalance() (sys/vfs/hammer/hammer_rebalance.c:53) is the backend of hammer rebalance / hammer cleanup (HAMMERIOC_REBALANCE, gated by caps_priv_check(SYSCAP_NOVFS_IOCTL) at hammer_ioctl.c:72). For every internal node visited on the way up, rebalance_node() (:266) locks the node's children (hammer_btree_lock_children, hammer_btree.c:2401) which stores each child's on-disk count straight into item->count (hammer_btree.c:2473) — node load validates only the CRC (hammer_load_node, hammer_ondisk.c:1313-1325); the count is an unvalidated int32_t (hammer_btree.h:225: "maximum 62 for INTERNAL, 63 for LEAF" — convention only).

rebalance_node then consumes it with no bounds check:

• count += item->count (:330) — signed-int accumulation of unvalidated int32s (sum can wrap negative; KKASSERT(avg_elms >= 0) :334 is INVARIANTS-only).
• avg_elms = howmany(count, lockroot.count) (:333) — avg can exceed the 63-element elms[] array of the 4096-byte kmalloc'd node copies (hammer_btree_lock_copy, hammer_btree.c:2505). The saturation clamp in the ioctl (:75-78) bounds rebal->saturation to [31,62]; it does NOT bound avg_elms when the forged counts already exceed saturation.
• pack loop for (i = 0; i < item->count; ++i) (:371) with elm = &base_item->copy->elms[base_count]; *elm = node->ondisk->elms[i]; (:406-407) — base_count runs up to avg_elms-1 → 64-byte-granule writes past the end of the 4KB copy (heap OOB WRITE), and node->ondisk->elms[i] for i ≥ 63 reads past the node's 4KB slice (OOB READ, in-buffer up to +16KB, then adjacent kernel heap).
• rebalance_closeout() stores copy->count = base_count > 63 (:541-543) which hammer_btree_sync_copy() (:2519-2537) writes back to the on-disk node — persistent filesystem corruption; every later btree_search into that leaf OOB-reads (KKASSERT at hammer_btree.c:1278 is INVARIANTS-only).
• the scan loop also consumes the count: cursor.index = count - 1 (:174) then elm = &cursor.node->ondisk->elms[cursor.index].leaf (:194) → OOB read whose 40 bytes are returned to the ioctl caller in rebal->key_cur (:195).

This is the DF-0776 family (missing on-disk count validation) at THIS file's sinks — DF-0776 covered the btree-core search/shift loops in hammer_btree.c; the rebalance pack loop is a distinct set of sinks that DF-0776's analysis did not cover and that needs its own validation (or a global load-time bounds check).

Threat model

Attacker crafts a HAMMER1 image (all CRCs recomputed — the CRC is not a MAC) and gets an administrator to mount it and run hammer rebalance / hammer cleanup on it (standard maintenance of attacker-supplied or attacker-corrupted volumes). The ioctl itself is privileged, so the unprivileged-user→root chain requires that admin interaction — same preconditions as DF-0776/DF-3167.

Reproduction (baseline, stock kernel)

1. mkimage.sh — 12G vn image, ~2500 dirents → B-Tree: root 0x2103f000, mid-level internals, 156 leaves (count=32 each).
2. patch_leaf_rebal walk — selects parent 0x21001000 (first child at the root) and its 14 conforming non-first leaf children (slice 0/1 of their 16K buffers, in-buffer elms[127]).
3. patch_leaf_rebal patch — forges count=128 on those 14 leaves, re-CRCs each node. Expected runtime math: 31 children, total = 1732 + 14128 = 2336 → avg_elms = ceil(2336/31) = 76 > 63.
4. trigger.sh — mount rw, hammer rebalance /mnt.

Observed (console_full.log / panic.txt):

rebalance_node: lockroot 0xfffff801166db308 count 31
rebalance_node: add count 32   (x17)
rebalance_node: add count 128  (x14)   <- unvalidated disk counts
rebalance_closeout: 8000000021000000: (count update) (parent bound update 1)
... (x31, includes the forged leaves 0x21004000 etc.)
panic: assertion "slab->magic == KMALLOC_SLAB_MAGIC && delta != slab->ncount"
       failed in _kfree_obj at /usr/src/sys/kern/kern_kmalloc.c:942
_kfree_obj() -> fdrop() -> closef() -> kern_close() -> syscall2()

The pack loop wrote elms[63..75] (13 granules x 64B = 832B) past the end of each of ~30 kmalloc'd 4KB node copies; the overflow smashed kmalloc slab metadata and was detected when an unrelated close() syscall freed a neighboring object. This is a direct, observed out-of-bounds kernel heap write, not a speculative one. The guest stayed in DDB (down).

Note: the panic interrupted the run before umount, so the >63 count sync-to-disk is evidenced by the (count update) closeout lines and by the fixed-kernel readback comparison below rather than by a baseline readback.

Primitive characterization (why this matters beyond the panic)

• Length control: overflow extent = avg_elms - 63 granules; the image author chooses each child count (up to INT32_MAX; with 62 children the sum itself overflows int at :330), so the write length is attacker-chosen (bytes to megabytes).
• Content control: the copied granules are node->ondisk->elms[i] — for i ≤ 62 these are the leaf's own on-image elements (fully attacker-controlled 64-byte blocks); for 63 ≤ i ≤ ~254 they are bytes of the same 16KB hammer buffer (on-image data of neighboring nodes or zeroed free space — still attacker-controlled at image-authoring time); beyond that, adjacent kernel heap (uncontrolled).
• Placement: sequential forward overflow from a 4096-byte kmalloc from the hammer m_misc pool; on this INVARIANTS guest the smash was detected by the slab-magic check; on production kernels there is no such check (silent corruption of adjacent heap objects).
• Gating: the sink requires a privileged ioctl caller (root runs hammer rebalance/hammer cleanup on the attacker's volume), so an unprivileged local user cannot reach it alone — the honest impact ceiling here is High (kernel heap corruption from a crafted volume during routine maintenance), matching DF-0776/DF-3167, not an unpriv→uid0 chain.

Fix validation

fix.diff adds two bounds checks using the existing hammer_node_max_elements() helper (hammer_btree.h:283):

1. scan loop (:146ff): validate cursor.node->ondisk->count and type before the count is consumed (:174/:194) — EIO aborts the scan.
2. rebalance_node accumulation (:355ff): validate each item->count before count += item->count — EIO, no packing.

Kernel #1 (Sep 7 13:18:26) built in-guest (make nativekernel KERNCONF=X86_64_GENERIC), same crafted image, same patch:

hammer_ioc_rebalance: rebalance: bad node 8000000021004000 type L count 128
Rebalance /mnt failed: Input/output error
Rebalance: 0 B-Tree nodes scanned, 0 rebalanced
VERIFY: 14 out-of-bounds counts found   <- ONLY the leaves we forged

vs. baseline (31 leaves rewritten with count=76 + slab-smash panic). Bad behavior GONE; guest healthy; clean umount.

Kernel references

• sys/vfs/hammer/hammer_rebalance.c:75-78 (saturation clamp — bounds saturation, not avg_elms)
• sys/vfs/hammer/hammer_rebalance.c:174,194-195 (scan consumes count)
• sys/vfs/hammer/hammer_rebalance.c:326-334 (unvalidated accumulation + int overflow)
• sys/vfs/hammer/hammer_rebalance.c:371,406-407 (copy loop: OOB read source + OOB write dest)
• sys/vfs/hammer/hammer_rebalance.c:541-543 (count > 63 synced to disk)
• sys/vfs/hammer/hammer_btree.c:2473 (item->count = child->ondisk->count)
• sys/vfs/hammer/hammer_ondisk.c:1313-1325 (load validates CRC only)
• sys/vfs/hammer/hammer_btree.h:225 (count: int32, max by convention)

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.