DF-3012 — hammer(1) in-memory record leaked when blockmap reservation

fails in hammer_ip_add_bulk() (+ same class in hammer_ip_add_direntry)

What (defect, code-certain)

hammer_ip_add_bulk() (sys/vfs/hammer/hammer_object.c:969-978) allocates a memory record (refcount 1) and then tries the direct-write blockmap reservation. On failure it releases the record without setting HAMMER_RECF_DELETED_FE:

record = hammer_alloc_mem_record(ip, 0);          /* ref = 1        */
record->resv = hammer_blockmap_reserve(...);
if (record->resv == NULL) {
        hdkprintf("reservation failed\n");
        hammer_rel_mem_record(record);            /* ref -> 0, LEAK */
        return (NULL);
}

hammer_rel_mem_record() (hammer_object.c:374-393) only destroys a record when HAMMER_RECF_DELETED_FE|BE or COMMITTED is set. A never-inserted, un-flagged record dropped to zero refs is freed by nobody — it is on no RB-tree, no target list, referenced by nothing. Both the record (~224 B, "HAMMER-others" malloc zone) and — in the sibling case — its kmalloc'd data leak permanently (until reboot; not reclaimed by sync, rm, or unmount logic that walks inodes).

The sibling site is hammer_ip_add_direntry() at hammer_object.c:711-716 (namekey-iteration exhaustion → ENOSPC): leaks record + HAMMER_ENTRY_SIZE(bytes) of direntry data.

Every comparable error path in the tree sets the flag before releasing — e.g. hammer_io_direct_write()'s failure path (hammer_io.c:1793-1795) does record->flags |= HAMMER_RECF_DELETED_FE; hammer_rel_mem_record(...) — these two sites were missed.

Trigger conditions (what Phase V established)

A record leaks once per hammer_blockmap_reserve() failure hit from hammer_vop_strategy_write() (hammer_vnops.c:3251 — every 16K-aligned buffer flush). The frontend hammer_checkspace() gate (hammer_vnops.c:605, estimate in hammer_blockmap.c:1267-1290) exists precisely to keep buffered writes from reaching an exhausted blockmap; its headroom is vfs.hammer.limit_dirtybufspace (default 49 MB = half the global dirty-buffer cap) + undo/slop reserves, and pending records are accounted into the estimate at strategy time. We could not make the reservation fail from an unprivileged user on default tuning despite: plain fills, 40,000-file tiny-file sprays, mmap/msync putpages (which go through VOP_WRITE and are gated too), and 6-parallel-writer unique-data races on both tmpfs-fast and virtio-slow backing storage, including with the gate headroom removed via vfs.hammer.limit_dirtybufspace=2M.

Realistic residual triggers (unproven in lab): freemap hammer_bread I/O errors inside hammer_blockmap_reserve() (failing/mismatched hardware), concurrent admin reblock/prune/mirror operations racing the frontend, multi-volume configurations, or deep drain-lag topologies (slow data path, fast writers) where the in-flight dirty backlog exceeds the reserve.

Reproduce (what the pack runs)

# in guest as root:  build + self-contained run (fresh 512M fs)
sh build.sh         # cc -O -o /tmp/op3012 /tmp/op3012.c -Wall
sh run.sh           # fills with urandom via 6 parallel 'nobody'
                    # writers on a nohistory fs, watches dmesg for
                    # "reservation failed" and the HAMMER-others zone

Expected on a vulnerable kernel when a reservation failure occurs: dmesg gains one hammer_ip_add_bulk: reservation failed line per leaked record and the HAMMER-others zone in vmstat -m grows monotonically, never shrinking after rm -rf + sync.

Observed on the lab guest (INVARIANTS kernel, default + reduced tuning): zero reservation failures across all attempts — the leak's trigger precondition could not be organically produced; see VERDICT.md for the full negative-result analysis and EXPERIMENTS.md for every attempt.

DF-3012 VERDICT — hammer(1) mem-record leak on blockmap reservation failure

Status: not_reproduced (defect code-certain; trigger precondition not reachable organically on the lab guest — see below). Guest stayed healthy throughout; no panics.

The defect (certain, by source)

hammer_ip_add_bulk() — sys/vfs/hammer/hammer_object.c:974-977:

if (record->resv == NULL) {
        hdkprintf("reservation failed\n");
        hammer_rel_mem_record(record);   /* no DELETED_FE -> never freed */
        return (NULL);
}

hammer_rel_mem_record() (hammer_object.c:374-393) destroys a record only when it carries HAMMER_RECF_DELETED_FE|BE or HAMMER_RECF_COMMITTED. The just-allocated record carries none and was never inserted into the inode's RB-tree, so dropping it to zero refs orphans it forever: ~224 B per record in the "HAMMER-others" malloc zone (hmp->m_misc), plus hammer_count_records never decrements. Sibling site: hammer_ip_add_direntry() :711-716 leaks record + entry data on the ENOSPC namekey-exhaustion path. The author's own error paths elsewhere set the flag first — hammer_io.c:1793-1795 (hammer_io_direct_write failure), hammer_inode.c:1360/1585/3183 — proving intent; these two sites were missed.

What Phase V did (guest: DragonFly 6.5-DEVELOPMENT #0, INVARIANTS)

The only caller is hammer_vop_strategy_write() (hammer_vnops.c:3251): every 16K-aligned buffer flush → hammer_ip_add_bulk() → hammer_blockmap_reserve(). Reservation failure modes (hammer_blockmap.c:419-623): zone wrap-twice → ENOSPC (real exhaustion), freemap layer1/layer2 hammer_bread I/O errors, hammer_bnew errors.

Attempts to drive an unprivileged user into a failing reservation, all with fresh HAMMER1 filesystems (vnconfig + newfs_hammer + mount, harness per DF-2999), attacker processes run as nobody:

1. Plain fill to ENOSPC + backlog sync (800M fs, 512M undo): write(2) stops at the checkspace gate; all dirty buffers flush cleanly. dmesg | grep -c "reservation failed" = 0; zone unchanged.
2. 40,000 tiny-file spray (unique contents, outruns the flusher's record accounting): all 40,000 created, 0 write failures, 0 reservation failures. The rsv_recs/rsv_databytes estimate (hammer_blockmap.c:1274) tracks real consumption once records land at strategy time.
3. mmap + msync(MS_SYNC) loops (512 iters × 3 rounds): every msync returned success. Investigation (probe.c) showed vnode_pager_putpages routes mmap flushes through VOP_WRITE (sys/vm/vnode_pager.c:768) — the same checkspace gate — so mmap cannot bypass the gate either; when the pager does fail it logs vnode_pager_putpages: I/O error 28 without touching add_bulk (observed in dmesg). (Side observation, out of file scope: msync(2) returned 0 while the pager write failed with ENOSPC — error not propagated to userspace.)
4. 6-parallel-writer unique-data races on tmpfs-backed (fast drain) and virtio/hammer2-backed (slow drain) images, nohistory mounts, pre-built random source for max dirty speed: fs filled to 100% cleanly every time; zero reservation failures.
5. Deterministic attempt with the gate headroom removed (sysctl vfs.hammer.limit_dirtybufspace=2097152, root-set; writes still by nobody): still zero failures — on this guest the buffer flush pipeline is effectively synchronous with write(2) (strategy runs inside the syscall path, so pending bytes are accounted before the next block's gate check), leaving no unaccounted in-flight window even at 2 MB headroom.

Why it still matters

hdkprintf("reservation failed") exists because Dillon expected the path to fire. Operationally it can fire when the freemap cannot be read (degraded hardware), during concurrent admin operations (reblock, prune, mirror, volume-add), or on drain-lag topologies where the in-flight buffer backlog exceeds the ~49 MB per-mount reserve against the ~103 MB global dirty cap (hammer's reserve is half the system-wide capacity — a designed-in 2× gap). When it fires, each failed buffer flush leaks one record permanently; a full-disk event under load leaks up to the backlog depth (hundreds of MB of kernel heap in the worst case on real hardware), unrecoverable without a reboot.

Exploit chain

None — kernel heap resource leak (availability), not memory corruption. No path to uid=0.

Fix

fix.diff — one line per site (matches hammer_io.c:1793's pattern):

    if (record->resv == NULL) {
        hdkprintf("reservation failed\n");
+       record->flags |= HAMMER_RECF_DELETED_FE;
        hammer_rel_mem_record(record);
        return(NULL);
    }

fix_status: not_testable — validating requires first reproducing a reservation failure, which Phase V could not trigger (see above); the diff is line-accurate against the read-only sys/ tree and mirrors the author's own correct pattern one call-layer down.

Kernel references

• sys/vfs/hammer/hammer_object.c:969-978 (leak, add_bulk)
• sys/vfs/hammer/hammer_object.c:711-716 (leak, add_direntry)
• sys/vfs/hammer/hammer_object.c:363-393 (rel_mem_record destroy-only-if-flagged)
• sys/vfs/hammer/hammer_vnops.c:3251 (sole caller, strategy write)
• sys/vfs/hammer/hammer_blockmap.c:419-623 (reserve failure modes)
• sys/vfs/hammer/hammer_blockmap.c:1267-1290 (checkspace estimate)
• sys/vfs/hammer/hammer_io.c:1793-1795 (correct sibling pattern)

DF-3012 experiment log (all Phase V attempts, chronological)

Guest: DragonFly 6.5-DEVELOPMENT #0 (Thu Jul 2 06:02:54 UTC 2026), X86_64_GENERIC, INVARIANTS, 6 vCPU KVM, 4 GB RAM. All attacker processes ran as nobody; root only set up vn/newfs/mount and took measurements.

Evidence channels used throughout: - dmesg | grep -c "reservation failed" — hdkprintf is ungated (hammer.h:1606), so every add_bulk failure prints exactly once. - vmstat -m | awk '/HAMMER-others/' — the m_misc zone where struct hammer_record is kmalloc'd; leaks here are permanent. - sysctl vfs.hammer.count_reservations.

E1 — plain fill + mmap/msync, 800M fs (512M undo), history mount

Fresh fs (264M usable / 38.8M avail). Stage A: 16K write loop until ENOSPC (59,473,920 bytes, errno 28). Stage B: mmap 64M file, 2000× dirty-page+msync(MS_SYNC) — all 2000 "succeeded". Zone 10→46 allocs (record churn only). 0 reservation failures. Follow-up probe (probe.c): msync "success" with zero hammer disk-write counter movement and a readback MATCH only for recently-touched pages; dmesg carried vnode_pager_putpages: I/O error 28 + residual I/O 262144 — putpages routes through VOP_WRITE (vnode_pager.c:768), i.e. the same checkspace gate; the pager's failure is not propagated to msync's return value. mmap is NOT a gate bypass.

E2 — same fs, PRNG-unique pages (defeat dedup), 3 rounds

16384 pages × 16K pushed per round "ok"; readback of last page: MISMATCH (data never reached the fs — gate rejected the putpages write, pages stayed dirty in the object). Zone unchanged, 0 failures. (Lesson: E1/E2's earlier "successes" were partly dedup artifacts — repeated/low-entropy blocks dedup in HAMMER1 and consume no bigblocks.)

E3 — tiny-file spray, 800M fs: 20,000 then 40,000 files (unique 1-15B)

40,000 files created, 0 write failures, 241M used / 22.8M avail left. Zone peaked 5.56K allocs during churn, settled 4.34K after rm+sync (pending-delete record drain), 0 reservation failures. The estimate's rsv_recs/rsv_databytes accounting keeps up with tiny-file metadata.

E4 — 256M fs with 64M undo (small-undo), nohistory

df showed avail=0 at format (df-side reserve ≥ free) — gate never passes; abandoned (also hit su-quoting issue, fixed with /tmp/writers.sh).

E5 — 6-parallel unique-data writers, 512M fs/64M undo on /root

(virtio/hammer2-backed image: slow drain) + /tmp/rand.bin source (fast dirtying), nohistory. Stage 1 filled 199M to 100%: 0 failures. rm freed (nohistory), stage 2 re-filled: 0 failures. Zone 10→150→169 allocs (churn). The buffer flush pipeline stayed synchronous with write(2) — no unaccounted in-flight window.

E6 — E5 with gate headroom removed: sysctl

vfs.hammer.limit_dirtybufspace=2M (root-set, runtime RW sysctl; writes still by nobody). Fill to 100%: STILL 0 reservation failures. On this guest, strategy runs inside the write(2) path before the next block's gate check, so pending reservations are always accounted and the blockmap never runs dry behind the gate.

Conclusion

The checkspace gate + synchronous strategy pipeline on this guest makes hammer_blockmap_reserve() == NULL unreachable organically. The leak defect itself is code-certain; residual real-world triggers: freemap bread I/O errors, concurrent admin ops (reblock/prune/mirror), drain-lag topologies where in-flight dirty exceeds the 49M per-mount reserve (hammer reserves half the global ~103M dirty cap), or reduced limit_dirtybufspace on systems where strategies DO lag.