β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-3067

PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β€” mount never terminates, unkillable CPU-burning kernel thread (livelock DoS)

Summary

For every version>=4 mount (stock images are v7) stage1 runs a reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record to seed hdr_seq. PAD records are explicitly exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan 'grossly ahead' bound is #if 0'd out at :312-335). An undo FIFO consisting solely of valid 8-byte PAD records (self-consistent head==tail, no CRC required) makes the mount spin forever. Stage2's extended-range scan (:652-670) has the same shape: it ends only when scan_offset == ext_offset, where ext_offset = the REDO_SYNC record's redo_offset - an unvalidated u64 that can be garbage and never match a record boundary. Mount of a crafted HAMMER image (root, or unpriv with vfs.usermount=1 - DF-0797/0798/0812/2594/3040 family); READ-ONLY mount suffices. The mount syscall never returns; the mounting thread spins at 100% CPU in kernel mode and kill -9 pends forever (no cancellation points), pinning a CPU and the vn device until reboot; each additional mount attempt stacks another spinning thread. Works on INVARIANTS and production kernels - no assertion involved. VERIFIED stock INVARIANTS guest: 47,039+ traced iterations still running, ring wrap observed at iterations 513 and 33281, all 32,769 ring offsets revisited (lap >= 2). Fix validated in-guest (one-lap walked-byte bound in both scans): identical image walks exactly one lap then fails cleanly EIO; stock mount/RW-cycle and hard-crash dirty-recovery regressions pass.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-3067 Β· 14 files
FileTypeDescriptionSize
padflood.c β€” 7.5 KB view raw
build.sh β€” 474 B view raw
run3067.sh β€” 1.3 KB view raw
README.md β€” 2.8 KB ↓ raw
VERDICT.md β€” 3.6 KB ↓ raw
run.log β€” 1.1 KB view raw
bootlog_walk_trace.log β€” 2.7 MB ↓ download
console_flood_sample.txt β€” 3.4 KB view raw
env.txt β€” 613 B view raw
fix.diff β€” 3.1 KB view raw
fix_build.log β€” 5.1 MB ↓ download
fix_run.log β€” 943 B view raw
manifest.json β€” 1.2 KB view raw
verdict.json β€” 5.3 KB view raw

DF-3067 β€” PAD-only undo FIFO makes hammer_recover_stage1's seqno backscan (and stage2's extended-range scan) unbounded β€” mount never terminates (livelock DoS)

Kernel: DragonFly 6.5-DEVELOPMENT #0 (X86_64_GENERIC, INVARIANTS), sys/vfs/hammer File under audit: sys/vfs/hammer/hammer_recover.c (pass 2). New finding β€” not DF-0812/DF-2594 (redo_data_bytes) nor DF-3040 (vol0_undo_array index).

Root cause (hammer_recover.c)

  • For every version β‰₯ 4 mount (stock images are version 7), stage1 runs a seqno backscan (:257-270): a for(;;) reverse FIFO walk whose only exits are an I/O/signature error or the first non-PAD record (to grab its hdr_seq). PAD records are explicitly exempt from the seqno discontinuity check (:266-269), and hammer_recover_scan_rev wraps the walk at the zone base (:789-790), making the FIFO a ring.
  • An undo FIFO consisting solely of valid 8-byte PAD records ({0xC84E, 0x8040, 8} β€” self-consistent head==tail, no CRC required per _hammer_check_signature:930) defeats both exits: no error, no non-PAD. The loop has no byte counter and no lap limit β†’ infinite loop at 100% CPU inside mount(2). The disabled #if 0 block at :312-335 (a "grossly beyond last_offset" bound for the sibling forward scan) shows the author considered bounding these walks β€” it is compiled out.
  • Same unbounded shape in stage2's extended-range scan (:652-670): it runs until scan_offset == ext_offset where ext_offset = hmp->recover_stage2_offset is REDO_SYNC.redo_offset β€” an unvalidated u64 from the crafted record (:423-424); a garbage value that is not a reachable record boundary plus an all-PAD gap = same infinite loop (RW mounts).

Threat model

Mount of a crafted HAMMER image (root, or unprivileged with vfs.usermount=1 β€” same precondition family as DF-0797/0798/0812/2594/3040). A read-only mount suffices for the stage1 variant. The mount syscall never returns; the mounting thread spins in kernel mode and cannot be killed (the loop has no cancellation points), pinning a CPU and the vn device forever. Works on INVARIANTS and production kernels (no assertion involved).

Reproduce

# guest (root): sh build.sh   (cc padflood.c; truncate 12G + newfs_hammer base)
sh run.sh
# expected:
#   padflood floods all undo bigblocks with PAD-8s, sets vol_version=6,
#   first=next=UNDO|0x1000, then:
#   MOUNTING ... (30s timeout) MOUNT_RC=124
#   ps: mount_hammer still present (unkillable, state R)
#   dmesg: thousands of "rev scan_offset 3000xxxxxxxxxxxx" lines cycling
#          down through the 1GB FIFO and wrapping - the unbounded walk
#   guest otherwise alive (other ssh commands keep working)

Success criterion: MOUNT_RC=124 + unkillable mount process + cyclic rev scan_offset debug flood + guest still up = reproduced (impact: dos).

VERDICT.md
↓ download raw

DF-3067 VERDICT β€” REPRODUCED (dos)

Status: reproduced / impact: dos / confidence: certain. Baseline verified on the stock INVARIANTS kernel (DragonFly 6.5-DEVELOPMENT #0, Thu Jul 2 06:02:54 UTC 2026); fix validated on the rebuilt kernel #1.

What was run

padflood backscan patches a real newfs_hammer image (12G sparse, undo FIFO 64 bigblocks, version 7β†’6): every byte of a 256KB undo ring (alloc_offset = UNDO|0x40000, undo bigblock 0 at phys 0x1088000) becomes a valid 8-byte PAD record {0xC84E,0x8040,8}, and vol0_blockmap[3].first_offset = next_offset = UNDO|0x1000. Mounting it (RO or RW) runs hammer_recover_stage1's version-4 seqno backscan (hammer_recover.c:257-270), which walks the FIFO backwards to find a non-PAD record to seed hdr_seq.

Why it hangs (path:line)

  • The backscan for(;;) has exactly two exits: an I/O/signature error, or a non-PAD record (:266-269) β€” PAD records are exempt from the seqno discontinuity check that bounds every other walk in this file.
  • hammer_recover_scan_rev wraps the walk at the zone base (:789-790: scan_offset == HAMMER_ENCODE_UNDO(0) β†’ alloc_offset), making the FIFO a ring; PAD-8 records are self-consistent head==tail (8 bytes, _hammer_check_signature:901-924,930,948-961 β€” no CRC needed), so every position validates and neither exit ever fires.
  • There is no byte counter and no lap limit in the loop (the sibling forward scan's "grossly ahead" bound is #if 0'd out at :312-335).

Baseline observations (stock kernel #0)

  • The mount never returns; the mounting thread spins at 100% CPU in kernel mode; kill -9 pends forever (no cancellation points) β€” the process cannot be reaped, vn0 stays configured.
  • With vfs.hammer.debug_general=0x80 the serial console trace (bootlog_walk_trace.log, 47,039 lines captured while still running) shows: first traced offset 3000000000001000 == the crafted first_offset; the walk descends to the zone base, wraps UP 3000000000000000 β†’ 300000000003fff8 at iterations 513 and 33281 (hammer_recover.c:789-790); 32,769 distinct offsets = the entire ring revisited β€” lap β‰₯ 2 at capture time, still descending.
  • Guest otherwise alive (fresh ssh sessions answered) β€” CPU burn + wedged mount + unrecoverable vn device; repeated mounts stack spinning threads.
  • Works on INVARIANTS and production kernels (no assertion involved); a read-only mount suffices (stage1 runs before the RW/RO distinction matters).

Stage2 variant (same root cause, not separately run): the extended-range scan :652-670 terminates only when scan_offset == ext_offset where ext_offset = REDO_SYNC.redo_offset β€” an unvalidated u64 (:423-424); a garbage value plus an all-PAD gap = the same infinite loop on RW mounts.

Fix validation (fix.diff, kernel #1: Sat Sep 5 22:10:54 UTC 2026)

One-lap walked bound added to both backscan loops. Identical PoC image on the patched kernel: the walk does exactly one lap (trace ends back at 3000000000001008) then recovery failure during seqno backscan β†’ Failed to recover HAMMER filesystem on mount, mount_hammer exits, guest up. Regressions: stock image mount/RW-write/sync/umount/remount cycle clean; hard-crash dirty recovery (reboot -q on mounted dirty fs) clean (recovery check seqno=0010014d … files intact).

Impact classification

Local DoS (livelock) from mount of a crafted image β€” root or vfs.usermount=1 (same precondition family as DF-0797/0798/0812/2594/3040); unkillable spinning kernel thread per mount attempt. Not memory corruption; no escalation path identified (the walk only reads FIFO buffers).

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

Identical PoC image on the patched kernel walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'); mount process exits and is reaped; guest stays up. Stock-image regressions pass: mount/RW-create/sync/umount/remount cycle clean; hard-crash dirty recovery (reboot -q on mounted dirty fs) clean - 'recovery check seqno=0010014d', mount RC=0, all crash-surviving files intact.

fix_run.log, fix_build.log (kernel #1 build), fix.diff
↓ fix.diffDragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #1: Sat Sep 5 22:10:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64

Confirmed kernel references

Detail

Evidence (decisive lines)

["bootlog_walk_trace.log: 47,039 'hammer_recover_scan_rev: rev scan_offset ...' console lines (still growing at capture), first offset 3000000000001000 == crafted first_offset", 'run.log: walk wrap analysis - offset jumps UP 3000000000000000 -> 300000000003fff8 at iterations 513 and 33281 (hammer_recover.c:789-790 ring wrap); 32,769 distinct offsets = entire ring revisited; guest alive (fresh ssh answered PROBE_OK)', "fix_run.log (kernel #1 + fix.diff): one lap traced, then 'recovery failure during seqno backscan' + 'Failed to recover HAMMER filesystem on mount', mount_hammer exits (zombie->reaped), guest up", 'fix_build.log: make nativekernel KERNCONF=X86_64_GENERIC completed; kernel #1 Sat Sep 5 22:10:54 UTC 2026', 'padflood.c: the forger - PAD-8 pattern {0xC84E,0x8040,8} flood + vol0_blockmap[3] patch (no vol_crc fixup needed, DF-3042)']

PoC changes

Pass-2 author wrote the PoC from scratch (padflood.c) on the DF-3040 harness pattern (newfs_hammer base image + vol0_blockmap patch, no CRC fixup). Two iterations: (1) both packs initially shipped /root/run.sh causing an scp collision - the first guest run accidentally executed DF-3068's desync script (which reproduced DF-3068 immediately); renamed to run3067.sh/run3068.sh. (2) Shrank the PAD ring from the stock 512MB FIFO to 256KB (alloc=UNDO|0x40000) so the cyclic walk's wrap is visible quickly in the debug trace.

Verified recommended fix

Bound the stage1 backscan and the stage2 extended-range scan to one FIFO lap (walked >= HAMMER_OFF_LONG_ENCODE(alloc_offset) -> EIO); see fix.diff

Verdict

REPRODUCED on the stock INVARIANTS kernel (fresh vm.sh reset with-src): mounting a crafted HAMMER image whose undo FIFO is a 256KB ring of valid 8-byte PAD records (vol_version=6, first=next=UNDO|0x1000, alloc=UNDO|0x40000) makes hammer_recover_stage1's version-4 seqno backscan (hammer_recover.c:257-270) spin forever - PAD records are exempt from the seqno discontinuity check (:266-269), hammer_recover_scan_rev wraps at the zone base making the FIFO a ring (:789-790), and the loop has no byte counter or lap limit (the sibling fwd-scan bound is #if 0'd at :312-335). Observed: mount never returns; 47,039+ 'rev scan_offset' debug iterations captured while still running, first traced offset == crafted first_offset, ring wrap 3000000000000000->300000000003fff8 at iterations 513 and 33281, all 32,769 ring offsets revisited (lap>=2); the mounting thread burns 100% CPU in kernel mode and kill -9 pends forever (no cancellation points); vn0 stays wedged; guest otherwise alive. Read-only mount suffices; works on INVARIANTS and production kernels. Stage2 variant shares the root cause: its extended-range scan (:652-670) terminates only at ext_offset = the unvalidated REDO_SYNC.redo_offset u64 (:423-424). FIXED and validated: fix.diff bounds both backscan loops to one FIFO lap; on the rebuilt kernel #1 the identical image walks exactly one lap then fails the mount cleanly ('recovery failure during seqno backscan' -> 'Failed to recover HAMMER filesystem on mount'), mount process exits, guest up; stock-image mount/RW cycle and hard-crash dirty recovery regressions all pass.