PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β mount never terminates, unkillable CPU-burning kernel thread (livelock DoS)
Summary
For every version>=4 mount (stock images are v7) stage1 runs a reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record to seed hdr_seq. PAD records are explicitly exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan 'grossly ahead' bound is #if 0'd out at :312-335). An undo FIFO consisting solely of valid 8-byte PAD records (self-consistent head==tail, no CRC required) makes the mount spin forever. Stage2's extended-range scan (:652-670) has the same shape: it ends only when scan_offset == ext_offset, where ext_offset = the REDO_SYNC record's redo_offset - an unvalidated u64 that can be garbage and never match a record boundary. Mount of a crafted HAMMER image (root, or unpriv with vfs.usermount=1 - DF-0797/0798/0812/2594/3040 family); READ-ONLY mount suffices. The mount syscall never returns; the mounting thread spins at 100% CPU in kernel mode and kill -9 pends forever (no cancellation points), pinning a CPU and the vn device until reboot; each additional mount attempt stacks another spinning thread. Works on INVARIANTS and production kernels - no assertion involved. VERIFIED stock INVARIANTS guest: 47,039+ traced iterations still running, ring wrap observed at iterations 513 and 33281, all 32,769 ring offsets revisited (lap >= 2). Fix validated in-guest (one-lap walked-byte bound in both scans): identical image walks exactly one lap then fails cleanly EIO; stock mount/RW-cycle and hard-crash dirty-recovery regressions pass.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3067 Β· 14 files| File | Type | Description | Size | |
|---|---|---|---|---|
| padflood.c | β | 7.5 KB | view raw | |
| build.sh | β | 474 B | view raw | |
| run3067.sh | β | 1.3 KB | view raw | |
| README.md | β | 2.8 KB | β raw | |
| VERDICT.md | β | 3.6 KB | β raw | |
| run.log | β | 1.1 KB | view raw | |
| bootlog_walk_trace.log | β | 2.7 MB | β download | |
| console_flood_sample.txt | β | 3.4 KB | view raw | |
| env.txt | β | 613 B | view raw | |
| fix.diff | β | 3.1 KB | view raw | |
| fix_build.log | β | 5.1 MB | β download | |
| fix_run.log | β | 943 B | view raw | |
| manifest.json | β | 1.2 KB | view raw | |
| verdict.json | β | 5.3 KB | view raw |
DF-3067 β PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β mount never terminates (livelock DoS)
Kernel: DragonFly 6.5-DEVELOPMENT #0 (X86_64_GENERIC, INVARIANTS), sys/vfs/hammer
File under audit: sys/vfs/hammer/hammer_recover.c (pass 2). New finding β
not DF-0812/DF-2594 (redo_data_bytes) nor DF-3040 (vol0_undo_array index).
Root cause (hammer_recover.c)
- For every version β₯ 4 mount (stock images are version 7), stage1 runs a
seqno backscan (
:257-270): afor(;;)reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record (to grab itshdr_seq). PAD records are explicitly exempt from the seqno discontinuity check (:266-269), andhammer_recover_scan_revwraps the walk at the zone base (:789-790), making the FIFO a ring. - An undo FIFO consisting solely of valid 8-byte PAD records
(
{0xC84E, 0x8040, 8}β self-consistent head==tail, no CRC required per_hammer_check_signature:930) defeats both exits: no error, no non-PAD. The loop has no byte counter and no lap limit β infinite loop at 100% CPU insidemount(2). The disabled#if 0block at:312-335(a "grossly beyond last_offset" bound for the sibling forward scan) shows the author considered bounding these walks β it is compiled out. - Same unbounded shape in stage2's extended-range scan (
:652-670): it runs untilscan_offset == ext_offsetwhereext_offset = hmp->recover_stage2_offsetisREDO_SYNC.redo_offsetβ an unvalidated u64 from the crafted record (:423-424); a garbage value that is not a reachable record boundary plus an all-PAD gap = same infinite loop (RW mounts).
Threat model
Mount of a crafted HAMMER image (root, or unprivileged with
vfs.usermount=1 β same precondition family as DF-0797/0798/0812/2594/3040).
A read-only mount suffices for the stage1 variant. The mount syscall
never returns; the mounting thread spins in kernel mode and cannot be killed
(the loop has no cancellation points), pinning a CPU and the vn device
forever. Works on INVARIANTS and production kernels (no assertion
involved).
Reproduce
# guest (root): sh build.sh (cc padflood.c; truncate 12G + newfs_hammer base) sh run.sh # expected: # padflood floods all undo bigblocks with PAD-8s, sets vol_version=6, # first=next=UNDO|0x1000, then: # MOUNTING ... (30s timeout) MOUNT_RC=124 # ps: mount_hammer still present (unkillable, state R) # dmesg: thousands of "rev scan_offset 3000xxxxxxxxxxxx" lines cycling # down through the 1GB FIFO and wrapping - the unbounded walk # guest otherwise alive (other ssh commands keep working)
Success criterion: MOUNT_RC=124 + unkillable mount process + cyclic
rev scan_offset debug flood + guest still up = reproduced (impact: dos).
DF-3067 VERDICT β REPRODUCED (dos)
Status: reproduced / impact: dos / confidence: certain. Baseline verified on the stock INVARIANTS kernel (DragonFly 6.5-DEVELOPMENT #0, Thu Jul 2 06:02:54 UTC 2026); fix validated on the rebuilt kernel #1.
What was run
padflood backscan patches a real newfs_hammer image (12G sparse, undo
FIFO 64 bigblocks, version 7β6): every byte of a 256KB undo ring
(alloc_offset = UNDO|0x40000, undo bigblock 0 at phys 0x1088000) becomes a
valid 8-byte PAD record {0xC84E,0x8040,8}, and
vol0_blockmap[3].first_offset = next_offset = UNDO|0x1000. Mounting it
(RO or RW) runs hammer_recover_stage1's version-4 seqno backscan
(hammer_recover.c:257-270), which walks the FIFO backwards to find a non-PAD
record to seed hdr_seq.
Why it hangs (path:line)
- The backscan
for(;;)has exactly two exits: an I/O/signature error, or a non-PAD record (:266-269) β PAD records are exempt from the seqno discontinuity check that bounds every other walk in this file. hammer_recover_scan_revwraps the walk at the zone base (:789-790:scan_offset == HAMMER_ENCODE_UNDO(0) β alloc_offset), making the FIFO a ring; PAD-8 records are self-consistent head==tail (8 bytes,_hammer_check_signature:901-924,930,948-961β no CRC needed), so every position validates and neither exit ever fires.- There is no byte counter and no lap limit in the loop (the sibling
forward scan's "grossly ahead" bound is
#if 0'd out at:312-335).
Baseline observations (stock kernel #0)
- The mount never returns; the mounting thread spins at 100% CPU in
kernel mode;
kill -9pends forever (no cancellation points) β the process cannot be reaped, vn0 stays configured. - With
vfs.hammer.debug_general=0x80the serial console trace (bootlog_walk_trace.log, 47,039 lines captured while still running) shows: first traced offset3000000000001000== the craftedfirst_offset; the walk descends to the zone base, wraps UP3000000000000000 β 300000000003fff8at iterations 513 and 33281 (hammer_recover.c:789-790); 32,769 distinct offsets = the entire ring revisited β lap β₯ 2 at capture time, still descending. - Guest otherwise alive (fresh ssh sessions answered) β CPU burn + wedged mount + unrecoverable vn device; repeated mounts stack spinning threads.
- Works on INVARIANTS and production kernels (no assertion involved); a read-only mount suffices (stage1 runs before the RW/RO distinction matters).
Stage2 variant (same root cause, not separately run): the extended-range
scan :652-670 terminates only when scan_offset == ext_offset where
ext_offset = REDO_SYNC.redo_offset β an unvalidated u64 (:423-424);
a garbage value plus an all-PAD gap = the same infinite loop on RW mounts.
Fix validation (fix.diff, kernel #1: Sat Sep 5 22:10:54 UTC 2026)
One-lap walked bound added to both backscan loops. Identical PoC image on
the patched kernel: the walk does exactly one lap (trace ends back at
3000000000001008) then recovery failure during seqno backscan β
Failed to recover HAMMER filesystem on mount, mount_hammer exits, guest
up. Regressions: stock image mount/RW-write/sync/umount/remount cycle clean;
hard-crash dirty recovery (reboot -q on mounted dirty fs) clean
(recovery check seqno=0010014d β¦ files intact).
Impact classification
Local DoS (livelock) from mount of a crafted image β root or
vfs.usermount=1 (same precondition family as DF-0797/0798/0812/2594/3040);
unkillable spinning kernel thread per mount attempt. Not memory corruption;
no escalation path identified (the walk only reads FIFO buffers).
Fix verification
fixedIdentical PoC image on the patched kernel walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'); mount process exits and is reaped; guest stays up. Stock-image regressions pass: mount/RW-create/sync/umount/remount cycle clean; hard-crash dirty recovery (reboot -q on mounted dirty fs) clean - 'recovery check seqno=0010014d', mount RC=0, all crash-surviving files intact.
fix_run.log, fix_build.log (kernel #1 build), fix.diff
Confirmed kernel references
Detail
Evidence (decisive lines)
["bootlog_walk_trace.log: 47,039 'hammer_recover_scan_rev: rev scan_offset ...' console lines (still growing at capture), first offset 3000000000001000 == crafted first_offset", 'run.log: walk wrap analysis - offset jumps UP 3000000000000000 -> 300000000003fff8 at iterations 513 and 33281 (hammer_recover.c:789-790 ring wrap); 32,769 distinct offsets = entire ring revisited; guest alive (fresh ssh answered PROBE_OK)', "fix_run.log (kernel #1 + fix.diff): one lap traced, then 'recovery failure during seqno backscan' + 'Failed to recover HAMMER filesystem on mount', mount_hammer exits (zombie->reaped), guest up", 'fix_build.log: make nativekernel KERNCONF=X86_64_GENERIC completed; kernel #1 Sat Sep 5 22:10:54 UTC 2026', 'padflood.c: the forger - PAD-8 pattern {0xC84E,0x8040,8} flood + vol0_blockmap[3] patch (no vol_crc fixup needed, DF-3042)']
PoC changes
Pass-2 author wrote the PoC from scratch (padflood.c) on the DF-3040 harness pattern (newfs_hammer base image + vol0_blockmap patch, no CRC fixup). Two iterations: (1) both packs initially shipped /root/run.sh causing an scp collision - the first guest run accidentally executed DF-3068's desync script (which reproduced DF-3068 immediately); renamed to run3067.sh/run3068.sh. (2) Shrank the PAD ring from the stock 512MB FIFO to 256KB (alloc=UNDO|0x40000) so the cyclic walk's wrap is visible quickly in the debug trace.
Verified recommended fix
Bound the stage1 backscan and the stage2 extended-range scan to one FIFO lap (walked >= HAMMER_OFF_LONG_ENCODE(alloc_offset) -> EIO); see fix.diff
Verdict
REPRODUCED on the stock INVARIANTS kernel (fresh vm.sh reset with-src): mounting a crafted HAMMER image whose undo FIFO is a 256KB ring of valid 8-byte PAD records (vol_version=6, first=next=UNDO|0x1000, alloc=UNDO|0x40000) makes hammer_recover_stage1's version-4 seqno backscan (hammer_recover.c:257-270) spin forever - PAD records are exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan bound is #if 0'd at :312-335). Observed: mount never returns; 47,039+ 'rev scan_offset' debug iterations captured while still running, first traced offset == crafted first_offset, ring wrap 3000000000000000->300000000003fff8 at iterations 513 and 33281, all 32,769 ring offsets revisited (lap>=2); the mounting thread burns 100% CPU in kernel mode and kill -9 pends forever (no cancellation points); vn0 stays wedged; guest otherwise alive. Read-only mount suffices; works on INVARIANTS and production kernels. Stage2 variant shares the root cause: its extended-range scan (:652-670) terminates only at ext_offset = the unvalidated REDO_SYNC.redo_offset u64 (:423-424). FIXED and validated: fix.diff bounds both backscan loops to one FIFO lap; on the rebuilt kernel #1 the identical image walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'), mount process exits, guest up; stock-image mount/RW cycle and hard-crash dirty recovery regressions all pass.
No comments yet.