DF-3068 — Undo-walk desync via PAD/fake-tail inconsistency → deterministic KKASSERT panic in hammer_recover_stage1 (INVARIANTS) / silent mis-recovery (production)

Kernel: DragonFly 6.5-DEVELOPMENT #0 (X86_64_GENERIC, INVARIANTS), sys/vfs/hammer File under audit: sys/vfs/hammer/hammer_recover.c (pass 2). New finding — distinct root cause from DF-0812/2594 (redo_data_bytes) and DF-3040 (vol0_undo_array).

Root cause (hammer_recover.c)

• For version < 4 filesystems the nominal undo range is taken verbatim from the crafted vol0_blockmap[3] (:225-226) — next_offset need not be a record boundary.
• _hammer_check_signature skips all head/tail agreement checks for PAD-8 records (:901-924) and re-derives the tail from head + hdr_size - 8 (:948-961), so the tail bytes at end_off-8 are never required to agree with the head the tail's tail_size points at when that head is a short PAD/DUMMY-style record. hammer_recover_scan_rev (:1023-1026, :814-815) then steps the walk back by the head's hdr_size while the tail's tail_size was the only thing anchoring validation — a crafted "fake tail" whose tail_size (24) points at a 24-byte DUMMY head makes one loop iteration consume 24 bytes where only 8 remain in bytes.
• Stage1's undo loop exits with bytes = -16 and KKASSERT(error || bytes == 0) at :460 fires → deterministic panic on INVARIANTS kernels. On production kernels the assert is absent: the mount silently continues, executes an UNDO/DUMMY record from outside the nominal undo range, then rewrites and flushes the volume-header blockmap (:481-488) — silent mis-recovery of the FIFO indices. (The copies themselves stay buffer-bounded — no memory unsafety; impact is DoS on INVARIANTS + wrong recovery semantics on production.)
• Sibling asserts reachable the same way: :394 (scan_offset != first_offset), stage2 :698/:716, and the rterm KKASSERT at :1286 when the stage2 forward walk desyncs against the reverse TERM scan.

Reproduce

# guest (root), after build.sh + run.sh of DF-3067 (vn0 wedged):
cp findings/poc/DF-3068/run.sh + padflood.c; sh build.sh (if needed)
sh run.sh          # padflood desync mode: vol_version=3, DUMMY@F-16 with
                  # fake tail at F (tail_size=24), first=F, next=F+8
# expected on the stock INVARIANTS guest:
#   Fatal trap ... assertion: error || bytes == 0 in hammer_recover_stage1
#   (guest wedged at db>)

Impact ceiling: deterministic mount-time panic (local DoS, crafted image; root or vfs.usermount=1). Low — INVARIANTS-only for the panic; production kernels get silent wrong recovery, no memory corruption (all undo copies are bounds-checked at :1053-1076).

DF-3068 VERDICT — REPRODUCED (panic / dos)

Status: reproduced / impact: panic / confidence: certain. Baseline verified on the stock INVARIANTS kernel (DragonFly 6.5-DEVELOPMENT #0); fix validated on the rebuilt kernel #1.

What was run

padflood desync patches the same stock newfs_hammer image to vol_version=3 (version<4 mounts take first_offset/next_offset verbatim from vol0_blockmap[3], hammer_recover.c:225-226) and writes a 24-byte CRC-valid DUMMY record at FIFO offset 0xFF0 whose tail bytes ({0xC74F, 0x0041, 24} at 0x1000) double as the reverse walk's next fake tail; first=UNDO|0x1000, next=UNDO|0x1008 → nominal undo range of 8 bytes.

Why it panics (path:line)

• hammer_check_tail_signature (:1023-1026) anchors the head at end_off - tail->tail_size — the tail's size — while hammer_recover_scan_rev (:815) steps the walk back by the head's hdr_size. For PAD/DUMMY-style short records _hammer_check_signature never requires the previous record's tail to agree with the head it points at (:901-924 skips the agreement block; :948-961 re-derives the tail from the head), so a crafted tail/size pair whose head is a 24-byte record makes one loop iteration consume 24 bytes when only 8 remain (bytes -= head->head.hdr_size at :431).
• The undo loop exits with bytes = 8 - 24 = -16 and KKASSERT(error || bytes == 0) at :460 fires → deterministic panic.

Baseline observation (stock kernel #0, clean run after fresh reset)

HAMMER(uabase) recovery undo  3000000000001000-3000000000001008 (8 bytes)(RW)
panic: assertion "error || bytes == 0" failed in hammer_recover_stage1 at /usr/src/sys/vfs/hammer/hammer_recover.c:460
hammer_recover_stage1() at hammer_recover_stage1+0x2cb
hammer_vfs_mount() at hammer_vfs_mount+0xa3f
sys_mount() at sys_mount+0x36b
Debugger("panic")  -> guest wedged at db>

(reproduced twice; the first, accidental run hit the identical panic — see panic_console_full.txt vs panic.txt.)

Non-INVARIANTS behavior (production kernels)

The assert is compiled out: the loop exits silently with bytes < 0, the mount succeeds, stage1 has executed a record from outside the nominal undo range (the fake-tail DUMMY), and it rewrites + flushes the volume-header blockmap (:481-488) — silent mis-recovery of FIFO indices. No memory unsafety (all hammer_recover_undo copies are clamped at :1053-1076), hence severity Low for this finding.

Fix validation (fix.diff, kernel #1)

Size guard added (hdr_size > bytes → EIO) before record execution. Identical PoC on the patched kernel: HAMMER(uabase) Corrupt UNDO record size 0018 at 3000000000000ff0 → mount: Input/output error, no panic, guest up. Regressions (stock mount/RW cycle, hard-crash dirty recovery) all clean.