PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β mount never terminates, unkillable CPU-burning kernel thread (livelock DoS)
Summary
For every version>=4 mount (stock images are v7) stage1 runs a reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record to seed hdr_seq. PAD records are explicitly exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan 'grossly ahead' bound is #if 0'd out at :312-335). An undo FIFO consisting solely of valid 8-byte PAD records (self-consistent head==tail, no CRC required) makes the mount spin forever. Stage2's extended-range scan (:652-670) has the same shape: it ends only when scan_offset == ext_offset, where ext_offset = the REDO_SYNC record's redo_offset - an unvalidated u64 that can be garbage and never match a record boundary. Mount of a crafted HAMMER image (root, or unpriv with vfs.usermount=1 - DF-0797/0798/0812/2594/3040 family); READ-ONLY mount suffices. The mount syscall never returns; the mounting thread spins at 100% CPU in kernel mode and kill -9 pends forever (no cancellation points), pinning a CPU and the vn device until reboot; each additional mount attempt stacks another spinning thread. Works on INVARIANTS and production kernels - no assertion involved. VERIFIED stock INVARIANTS guest: 47,039+ traced iterations still running, ring wrap observed at iterations 513 and 33281, all 32,769 ring offsets revisited (lap >= 2). Fix validated in-guest (one-lap walked-byte bound in both scans): identical image walks exactly one lap then fails cleanly EIO; stock mount/RW-cycle and hard-crash dirty-recovery regressions pass.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3067 Β· 14 files| File | Type | Description | Size | |
|---|---|---|---|---|
| padflood.c | β | 7.5 KB | view raw | |
| build.sh | β | 165 B | view raw | |
| run3067.sh | β | 1.3 KB | view raw | |
| README.md | β | 2.8 KB | β raw | |
| VERDICT.md | β | 3.9 KB | β raw | |
| run.log | β | 1.1 KB | view raw | |
| bootlog_walk_trace.log | β | 2.7 MB | β download | |
| console_flood_sample.txt | β | 3.4 KB | view raw | |
| env.txt | β | 613 B | view raw | |
| fix.diff | β | 3.1 KB | view raw | |
| fix_build.log | β | 5.1 MB | β download | |
| fix_run.log | β | 943 B | view raw | |
| manifest.json | β | 1.2 KB | view raw | |
| verdict.json | β | 5.2 KB | view raw |
DF-3067 β PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β mount never terminates (livelock DoS)
Kernel: DragonFly 6.5-DEVELOPMENT #0 (X86_64_GENERIC, INVARIANTS), sys/vfs/hammer
File under audit: sys/vfs/hammer/hammer_recover.c (pass 2). New finding β
not DF-0812/DF-2594 (redo_data_bytes) nor DF-3040 (vol0_undo_array index).
Root cause (hammer_recover.c)
- For every version β₯ 4 mount (stock images are version 7), stage1 runs a
seqno backscan (
:257-270): afor(;;)reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record (to grab itshdr_seq). PAD records are explicitly exempt from the seqno discontinuity check (:266-269), andhammer_recover_scan_revwraps the walk at the zone base (:789-790), making the FIFO a ring. - An undo FIFO consisting solely of valid 8-byte PAD records
(
{0xC84E, 0x8040, 8}β self-consistent head==tail, no CRC required per_hammer_check_signature:930) defeats both exits: no error, no non-PAD. The loop has no byte counter and no lap limit β infinite loop at 100% CPU insidemount(2). The disabled#if 0block at:312-335(a "grossly beyond last_offset" bound for the sibling forward scan) shows the author considered bounding these walks β it is compiled out. - Same unbounded shape in stage2's extended-range scan (
:652-670): it runs untilscan_offset == ext_offsetwhereext_offset = hmp->recover_stage2_offsetisREDO_SYNC.redo_offsetβ an unvalidated u64 from the crafted record (:423-424); a garbage value that is not a reachable record boundary plus an all-PAD gap = same infinite loop (RW mounts).
Threat model
Mount of a crafted HAMMER image (root, or unprivileged with
vfs.usermount=1 β same precondition family as DF-0797/0798/0812/2594/3040).
A read-only mount suffices for the stage1 variant. The mount syscall
never returns; the mounting thread spins in kernel mode and cannot be killed
(the loop has no cancellation points), pinning a CPU and the vn device
forever. Works on INVARIANTS and production kernels (no assertion
involved).
Reproduce
# guest (root): sh build.sh (cc padflood.c; truncate 12G + newfs_hammer base) sh run.sh # expected: # padflood floods all undo bigblocks with PAD-8s, sets vol_version=6, # first=next=UNDO|0x1000, then: # MOUNTING ... (30s timeout) MOUNT_RC=124 # ps: mount_hammer still present (unkillable, state R) # dmesg: thousands of "rev scan_offset 3000xxxxxxxxxxxx" lines cycling # down through the 1GB FIFO and wrapping - the unbounded walk # guest otherwise alive (other ssh commands keep working)
Success criterion: MOUNT_RC=124 + unkillable mount process + cyclic
rev scan_offset debug flood + guest still up = reproduced (impact: dos).
DF-3067 VERDICT β REPRODUCED (dos)
Status: reproduced / impact: dos / confidence: certain. Baseline verified on the stock INVARIANTS kernel (DragonFly 6.5-DEVELOPMENT #0, Thu Jul 2 06:02:54 UTC 2026); fix validated on the rebuilt kernel #1.
What was run
padflood backscan patches a real newfs_hammer image (12G sparse, undo
FIFO 64 bigblocks, version 7β6): every byte of a 256KB undo ring
(alloc_offset = UNDO|0x40000, undo bigblock 0 at phys 0x1088000) becomes a
valid 8-byte PAD record {0xC84E,0x8040,8}, and
vol0_blockmap[3].first_offset = next_offset = UNDO|0x1000. Mounting it
(RO or RW) runs hammer_recover_stage1's version-4 seqno backscan
(hammer_recover.c:257-270), which walks the FIFO backwards to find a non-PAD
record to seed hdr_seq.
Why it hangs (path:line)
- The backscan
for(;;)has exactly two exits: an I/O/signature error, or a non-PAD record (:266-269) β PAD records are exempt from the seqno discontinuity check that bounds every other walk in this file. hammer_recover_scan_revwraps the walk at the zone base (:789-790:scan_offset == HAMMER_ENCODE_UNDO(0) β alloc_offset), making the FIFO a ring; PAD-8 records are self-consistent head==tail (8 bytes,_hammer_check_signature:901-924,930,948-961β no CRC needed), so every position validates and neither exit ever fires.- There is no byte counter and no lap limit in the loop (the sibling
forward scan's "grossly ahead" bound is
#if 0'd out at:312-335).
Baseline observations (stock kernel #0)
- The mount never returns; the mounting thread spins at 100% CPU in
kernel mode;
kill -9pends forever (no cancellation points) β the process cannot be reaped, vn0 stays configured. - With
vfs.hammer.debug_general=0x80the serial console trace (bootlog_walk_trace.log, 47,039 lines captured while still running) shows: first traced offset3000000000001000== the craftedfirst_offset; the walk descends to the zone base, wraps UP3000000000000000 β 300000000003fff8at iterations 513 and 33281 (hammer_recover.c:789-790); 32,769 distinct offsets = the entire ring revisited β lap β₯ 2 at capture time, still descending. - Guest otherwise alive (fresh ssh sessions answered) β CPU burn + wedged mount + unrecoverable vn device; repeated mounts stack spinning threads.
- Works on INVARIANTS and production kernels (no assertion involved); a read-only mount suffices (stage1 runs before the RW/RO distinction matters).
Stage2 variant (same root cause, not separately run): the extended-range
scan :652-670 terminates only when scan_offset == ext_offset where
ext_offset = REDO_SYNC.redo_offset β an unvalidated u64 (:423-424);
a garbage value plus an all-PAD gap = the same infinite loop on RW mounts.
Fix validation (fix.diff, kernel #1: Sat Sep 5 22:10:54 UTC 2026)
One-lap walked bound added to both backscan loops. Identical PoC image on
the patched kernel: the walk does exactly one lap (trace ends back at
3000000000001008) then recovery failure during seqno backscan β
Failed to recover HAMMER filesystem on mount, mount_hammer exits, guest
up. Regressions: stock image mount/RW-write/sync/umount/remount cycle clean;
hard-crash dirty recovery (reboot -q on mounted dirty fs) clean
(recovery check seqno=0010014d β¦ files intact).
Impact classification
Local DoS (livelock) from mount of a crafted image β root or
vfs.usermount=1 (same precondition family as DF-0797/0798/0812/2594/3040);
unkillable spinning kernel thread per mount attempt. Not memory corruption;
no escalation path identified (the walk only reads FIFO buffers).
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedIdentical PoC image on the patched kernel walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'); mount process exits and is reaped; guest stays up. Stock-image regressions pass: mount/RW-create/sync/umount/remount cycle clean; hard-crash dirty recovery (reboot -q on mounted dirty fs) clean - 'recovery check seqno=0010014d', mount RC=0, all crash-surviving files intact.
fix_run.log, fix_build.log (kernel #1 build), fix.diff
Confirmed kernel references
Detail
Evidence (decisive lines)
["bootlog_walk_trace.log: 47,039 'hammer_recover_scan_rev: rev scan_offset ...' console lines (still growing at capture), first offset 3000000000001000 == crafted first_offset", 'run.log: walk wrap analysis - offset jumps UP 3000000000000000 -> 300000000003fff8 at iterations 513 and 33281 (hammer_recover.c:789-790 ring wrap); 32,769 distinct offsets = entire ring revisited; guest alive (fresh ssh answered PROBE_OK)', "fix_run.log (kernel #1 + fix.diff): one lap traced, then 'recovery failure during seqno backscan' + 'Failed to recover HAMMER filesystem on mount', mount_hammer exits (zombie->reaped), guest up", 'fix_build.log: make nativekernel KERNCONF=X86_64_GENERIC completed; kernel #1 Sat Sep 5 22:10:54 UTC 2026', 'padflood.c: the forger - PAD-8 pattern {0xC84E,0x8040,8} flood + vol0_blockmap[3] patch (no vol_crc fixup needed, DF-3042)']
PoC changes
Pass-2 author wrote the PoC from scratch (padflood.c) on the DF-3040 harness pattern (newfs_hammer base image + vol0_blockmap patch, no CRC fixup). Two iterations: (1) both packs initially shipped /root/run.sh causing an scp collision - the first guest run accidentally executed DF-3068's desync script (which reproduced DF-3068 immediately); renamed to run3067.sh/run3068.sh. (2) Shrank the PAD ring from the stock 512MB FIFO to 256KB (alloc=UNDO|0x40000) so the cyclic walk's wrap is visible quickly in the debug trace.
Verified recommended fix
Bound the stage1 backscan and the stage2 extended-range scan to one FIFO lap (walked >= HAMMER_OFF_LONG_ENCODE(alloc_offset) -> EIO); see fix.diff
Verdict
REPRODUCED on the stock INVARIANTS kernel (fresh vm.sh reset with-src): mounting a crafted HAMMER image whose undo FIFO is a 256KB ring of valid 8-byte PAD records (vol_version=6, first=next=UNDO|0x1000, alloc=UNDO|0x40000) makes hammer_recover_stage1's version-4 seqno backscan (hammer_recover.c:257-270) spin forever - PAD records are exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan bound is #if 0'd at :312-335). Observed: mount never returns; 47,039+ 'rev scan_offset' debug iterations captured while still running, first traced offset == crafted first_offset, ring wrap 3000000000000000->300000000003fff8 at iterations 513 and 33281, all 32,769 ring offsets revisited (lap>=2); the mounting thread burns 100% CPU in kernel mode and kill -9 pends forever (no cancellation points); vn0 stays wedged; guest otherwise alive. Read-only mount suffices; works on INVARIANTS and production kernels. Stage2 variant shares the root cause: its extended-range scan (:652-670) terminates only at ext_offset = the unvalidated REDO_SYNC.redo_offset u64 (:423-424). FIXED and validated: fix.diff bounds both backscan loops to one FIFO lap; on the rebuilt kernel #1 the identical image walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'), mount process exits, guest up; stock-image mount/RW cycle and hard-crash dirty recovery regressions all pass.
No comments yet.