DF-3215 — HAMMER1 UNDO/REDO FIFO exhaustion panic in hammer_generate_redo

What

hammer_generate_redo() (sys/vfs/hammer/hammer_redo.c:90-91) panics the kernel when the UNDO/REDO FIFO has less than len + 48KB free:

    if (hammer_undo_space(trans) < len + HAMMER_BUFSIZE*3)
        hpanic("insufficient UNDO/REDO FIFO space for redo!");

There is no waiting/throttling path for REDO generation. The frontend write() lays REDO records (with full data payloads) into the FIFO at memory speed under undo_lock; space is reclaimed only when the flusher completes a full finalize cycle, and the on-disk first_offset that governs hammer_undo_space() (hammer_undo.c:477-498) lags at least one full cycle behind (hammer_flusher.c:793-836). Default vfs.hammer.fsync_mode = 3 (hammer_vfsops.c:100) makes fsync() take the REDO fast path, which resets the 4MB per-inode REDO budget (hammer_vnops.c:239) forever — so an unprivileged write()+fsync() loop lays unbounded REDO records between reclaims.

The trailing REDO_SYNC emission (hammer_redo.c:288-291) runs after the main record has been laid and next_offset advanced, so the panic can fire there even when the caller's own record fit.

Reproduce (on the audit guest)

# one-time setup (root): HAMMER1 fs with the minimum 8MB UNDO/REDO FIFO
# (newfs_hammer -f -u 8m — a supported configuration; mkhammer only warns)
sh /tmp/df3215/setup_v2.sh        # vn0 image + newfs_hammer + mount /mnt/h1

# trigger (unprivileged):
su -m nobody -c "/tmp/df3215/redo_fill /mnt/h1/scratch 1 1048576 30"

Expected output (stock kernel, unprivileged)

Guest panics after ~6 rounds (~6MB of REDO records into the 8MB FIFO):

panic: hammer_generate_redo: insufficient UNDO/REDO FIFO space for redo!
hammer_generate_redo() at hammer_generate_redo+0x575
hammer_generate_redo() at hammer_generate_redo+0x575   <- trailing SYNC emit
hammer_vop_write() at hammer_vop_write+0x675
vn_write() -> kern_pwritev()

SSH dies, guest sits in ddb. See panic.txt, run_v2.log.

Scope / variants

• V2 (deterministic, reproduced): 8MB FIFO (newfs_hammer minimum with -f).
• V1 (default 512MB FIFO): NOT reproduced on the QEMU guest — the page-cached virtio flusher reclaimed fast enough (3.84GB of REDO laid, no overflow). On real (especially spinning) storage the same race window exists, but that claim is not demonstrated here.

Fix

fix.diff replaces the hpanic with a graceful ENOSPC failure and adds a FIFO-pressure gate to the write path that disables REDO (the existing degradation idiom) until space returns. Validated: patched kernel does not panic; writes succeed with REDO disabled; mount stays healthy.

DF-3215 — VERDICT

Finding: hammer_generate_redo() panics the kernel when the UNDO/REDO FIFO is exhausted — reachable by an unprivileged write()/fsync() loop on any HAMMER (v4+) filesystem whose UNDO/REDO FIFO is small enough for the frontend to fill between two flusher finalize cycles.

Status: REPRODUCED (panic / unprivileged local DoS), fix validated.

How it reproduces (baseline kernel #0, stock INVARIANTS)

1. setup_v2.sh (root, one-time, realistic admin action): a 12GB vn image formatted newfs_hammer -f -u 8m — the minimum UNDO/REDO FIFO size mkhammer supports (sbin/hammer/ondisk.c:format_undomap, min = one 8MB big-block; mkhammer merely warns "may lead to VFS panics") — mounted at /mnt/h1 with a 1777 scratch dir.
2. redo_fill as nobody: first fsync() per file enables HAMMER_INODE_REDO (hammer_vnops.c:253-259); then repeated (write 1MB, fsync) pairs. Each fast-fsync (default vfs.hammer.fsync_mode=3, hammer_vfsops.c:100) resets the 4MB per-inode REDO budget (hammer_vnops.c:239), so the write path keeps calling hammer_generate_redo() (hammer_vnops.c:746), which lays REDO records with the full data payload into the FIFO at memory speed (hammer_redo.c:97-277).
3. Space is only reclaimed when the flusher completes a full finalize cycle, and the on-disk first_offset that hammer_undo_space() uses (hammer_undo.c:477-498) lags at least one full cycle behind (hammer_flusher.c:736-800, comment "REDOs can be added at any time"). After ~6 rounds (~6MB of REDO into the 8MB FIFO) the check at hammer_redo.c:90 trips:

panic: hammer_generate_redo: insufficient UNDO/REDO FIFO space for redo!
hammer_generate_redo() at hammer_generate_redo+0x575
hammer_generate_redo() at hammer_generate_redo+0x575   <- trailing REDO_SYNC
hammer_vop_write() at hammer_vop_write+0x675
vop_write() -> vn_write() -> kern_pwritev()

The doubled frame shows the panic fired in the trailing REDO_SYNC emission (hammer_redo.c:288-291 → hammer_generate_redo_sync → hammer_generate_redo) — that path runs after the caller's record was already laid and next_offset advanced, so even a caller whose own record fit can panic there. Guest sat in ddb; ssh dead. Evidence: panic.txt, run.log.

Scope limits (honest negatives)

• Default 512MB FIFO: NOT reproduced on this guest (run.3.log). 2.56GB of REDO laid by 4 parallel unprivileged writers, plus a second attempt with 4GB of co-resident dirty data stalling finalize reclaim — the QEMU page-cached virtio flusher always reclaimed in time (vfs.hammer.stats_redo reached 3.84GB without exhaustion). On real storage (esp. HDD sync latencies) the same window exists, but that is not demonstrated here. The demonstrated precondition is a small-undo filesystem (8MB, mkhammer-supported).
• Not memory corruption: the primitive is a deterministic hpanic, i.e. denial of service. No escalation path.

Why this is a real defect

• There is no waiting/throttling mechanism anywhere on the REDO generation path: the only backpressure is the panic itself. Compare hammer_generate_undo() (hammer_undo.c:136-137) — same panic, but its callers are flusher-side; the REDO one is frontend write()-reachable.
• mkhammer permits the triggering configuration (-f -u 8m) and the kernel does not reject small UNDO FIFOs at mount; the resulting panic is triggered by an unprivileged user, not by the admin who mounted.
• REDO is an optional optimization (disabled by hammer_skip_redo, degraded per-inode when the 4MB budget is exceeded, hammer_vnops.c:751-753) — panic is never the correct degradation.

Fix validation (kernel #1, built in-guest from fix.diff)

• Baseline reproduced (panic, above). fix.diff applied to the guest's /usr/src (repo sys/ untouched), make nativekernel KERNCONF=X86_64_GENERIC + installkernel, reboot.
• Exact same run.sh re-run: all 30 rounds complete in ~1s, no panic, guest up, mount healthy (run.2.log). The FIFO-pressure gate in hammer_vop_write disables REDO (the existing degradation idiom) once free FIFO space drops below 2x the per-inode budget; the redo.c change (ENOSPC instead of hpanic) backstops the ungated callers (REDO_TRUNC at hammer_vnops.c:2256, backend TERM_WRITE/TERM_TRUNC at hammer_inode.c:2856/3038, and the trailing SYNC emission).
• Note: on an 8MB FIFO the gate (2x4MB) disables REDO before exhaustion, so the ENOSPC backstop itself was not directly exercised by this run; it is a 10-line early-return on the failure path.

Kernel references

• sys/vfs/hammer/hammer_redo.c:87-91 (the hpanic; lock+check)
• sys/vfs/hammer/hammer_redo.c:97-277 (unthrottled record emission loop)
• sys/vfs/hammer/hammer_redo.c:288-291 (trailing REDO_SYNC emission — where the panic actually fired)
• sys/vfs/hammer/hammer_undo.c:477-512 (space = in-core next_offset − on-disk first_offset)
• sys/vfs/hammer/hammer_flusher.c:736-800 (reclaim lag; "REDOs can be added at any time")
• sys/vfs/hammer/hammer_vnops.c:225-246,253-259,743-754 (REDO fast-fsync enable/reset/generate)
• sys/vfs/hammer/hammer_vfsops.c:93,100 (limit_redo=4MB, fsync_mode=3 defaults)
• sbin/hammer/ondisk.c:745-780 (mkhammer min 8MB undo FIFO with -f)

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.