DF-2642 — zero_write() drops hammer2_chain_delete() failure: zero-block overwrite silently succeeds while the old data chain survives (stale-data resurrection)

What

zero_write() (sys/vfs/hammer2/hammer2_strategy.c:1298-1302) ignores the return value of hammer2_chain_delete(). When the deletion fails — reproducibly at the ENOSPC wall of a full PFS, where the parent COW allocation in _hammer2_chain_delete_helper (hammer2_chain.c:3542, hammer2_chain_modify(parent, ...)) fails — the error is dropped, *errorp stays 0, the strategy write completes the bio with bp->b_error = 0, and write(2) of an all-zero 64KB block reports full success.

The old DATA chain remains intact in the topology. The zero-filled logical buffer sits clean in the buffer cache, so immediate re-reads show zeros and the application believes the overwrite happened. Once that clean buffer is recycled (memory pressure / cache churn — observed naturally at the wall), reads go back through the chain and return the PRE-OVERWRITE content. No error is ever reported for the reads; the same offset was observed oscillating between zeros and the old pattern across successive reads.

This is the "secure overwrite" use-case (dd if=/dev/zero of=file conv=notrunc, scrub-before-delete, crypto wiping) silently not happening.

Distinct from DF-2633 (backend allocation failure for new data blocks + inode inserts): here nothing new is allocated — an existing chain deletion fails and the old content stays live. DF-2633's fix (inode error state fed by the strategy-write failure branch) does NOT cover this site: the error never reaches that branch, because zero_write swallows it before the feed.

Reproduce (stock INVARIANTS kernel #0)

# on the DragonFly guest, as root
cc -O2 -o zero2642 zero2642.c
sh run2642-in-guest   # or drive stages per run.sh

Host-driven (what run.sh does, stage by stage):

1. 64MB vn-over-tmpfs image, newfs_hammer2, mount at /mnt/h42. ./zero2642 setup /mnt/h42 64 — 64 victim files, each 128KB: block0+block1 filled with DF2642-Vnn-Bx-OLD-... patterns, fsync'd.
2. ./zero2642 fill /mnt/h42 90000 — fast incompressible fill until free <= 8MB (3 polls).
3. ./zero2642 overwrite /mnt/h42 64 — v0 as healthy control, then waits for free <= 12MB, forks an mmap drain child (mmap-dirties free+8MB of drain.bin — mmap putpages bypass the vop_write frontend ENOSPC gate, so the freemap drains to true exhaustion while pmp->free_nominal is still cached high), then round-robin hammers zero-pwrites over all remaining victims every ~3ms.
4. Readback (cached view), churnroot /usr/src (root-fs reads cycle the buffer cache without touching the wedged mount), hog, readback (decisive).

Expected (stock) — observed in run1 (findings/poc/DF-2642/run1/)

• OV_OK i=43..63 phase=W rot=0 fsync=-1 errno=32 free=4194304: pwrite() returned 65536 (success) for 21 victims at the wall (fsync returns -1 with the raw HAMMER2 NOSPC code 32 leaking as an errno — itself an unmapped-error bug).
• stage5 (cached readback): READBACK_DONE old=0 zeros=64 — the lie is total.
• stage7 (after churnroot): READBACK_DONE old=1 zeros=57 mixed=6 — v43/46/47/51/53/54/58 return pre-overwrite content. Hexdumps: v54 block1 = 44 46 32 36 34 32 2d 56 35 34 2d 42 31 2d 4f 4c 44... ("DF2642-V54-B1-OLD-", pure OLD); v43 = zeros prefix + pattern tail (partial page recycling).
• The state OSCILLATES: minutes later the same read returned zeros (stage9), and a subsequent direct read returned the OLD pattern again — successive reads of one offset return different content with no error.
• Console: xop_strategy_write: error 32 loff=0000000031c0000... storm from the drain buffers (backend NOSPC active at the same moment), plus hammer2_chain_create_indirect: error 00000020 No Space on Device and hammer2: backend unable to insert inode ....
• Controls: v0-v42 (overwritten while the backend was healthy) never flip — zeros before and after churn.

Reproducibility: in-window silent acceptance (OV_OK + fsync -1/32) occurred in 4/4 full runs; the post-eviction stale read manifested in run1 (7/21 in-window victims) — the delete failure depends on the freemap's radix availability at the instant of the parent COW, which is racy. The mechanism is source-proven either way.

Fix validation (kernel #2 + fix.diff)

With *errorp |= hammer2_chain_delete(...) propagated, the strategy completion error branch runs for failed zero-deletes: the logical buffer is completed with B_ERROR/EIO and the console gains an explicit xop_strategy_write: error 32 line for the zero block. Criterion: on the fixed kernel the stage5 cached readback reports err>0 (EIO) for failed-delete victims instead of clean zeros (see fix_validation.log).

Files

zero2642.c        multi-stage PoC (setup/fill/overwrite/churn/hog/readback)
run.sh            host-side stage driver (vm.sh run_root per stage)
walk2642.py       offline media topology walker (informational)
run1/…run4/       per-attempt stage logs + console captures
fix.diff          git-applyable one-hunk fix
VERDICT.md        full narrative verdict
manifest.json / verdict.json

DF-2642 — VERDICT

Finding: zero_write() in sys/vfs/hammer2/hammer2_strategy.c ignores the return value of hammer2_chain_delete(). When the deletion fails (parent COW allocation failure — ENOSPC wall), the all-zero block overwrite is reported to userland as a complete success (write(2) returns 65536, buffer completed with b_error=0), the old data chain survives in the live topology, and reads resurrect the pre-overwrite content once the clean zero-filled logical buffer is recycled.

Classification: REPRODUCED (silent non-durable overwrite + stale-data resurrection + read oscillation on the live mount), fix FIXED.

1. Where the error is lost (source trace)

• All-zero 64KB writes funnel to zero_write() (hammer2_strategy.c:1271-1309) from both compression paths (hammer2_zero_check_and_write :1217, hammer2_compress_and_write :931) whenever check_algo != HAMMER2_CHECK_NONE (default XXHASH64) and test_block_zeros() passes.
• For a real (non-hole) DATA chain the entire overwrite is implemented as a deletion: strategy.c:1299-1300 hammer2_chain_delete(*parentp, chain, mtid, HAMMER2_DELETE_PERMANENT); — return value discarded, *errorp untouched, ++hammer2_iod_file_wzero counts a success regardless.
• hammer2_chain_delete → _hammer2_chain_delete_helper (hammer2_chain.c:3542) fails at hammer2_chain_modify(parent, ...) when the freemap cannot allocate the parent COW (hammer2_freemap_try_alloc → HAMMER2_ERROR_ENOSPC, hammer2_freemap.c:358/396). On a full PFS this is exactly the failure mode quantified by DF-2633.
• With *errorp == 0 the xop feeds error 0 (strategy.c:632), collect returns success/ENOENT, and the completion branch (strategy.c:670-674) does bp->b_resid = 0; bp->b_error = 0; biodone(bio) — the overwrite is now a clean, cached, "successful" buffer.
• The still-intact chain keeps the old data_off; after the clean buffer is recycled, the read path (hammer2_xop_strategy_read → hammer2_strategy_read_completion, COMP_NONE bcopy(data, bp->b_data, focus->bytes) at :487) serves the old block. No error is ever attached to any subsequent operation.

Why DF-2633's fix does not cover it: their deferred ip->error is fed from the strategy-write completion failure branch (strategy.c:670-681) and the inode-insert paths — the zero_write site never produces an error to feed it with.

2. Observed (stock kernel #0, INVARIANTS, run1)

64MB vn-over-tmpfs hammer2, 64 pattern victims, fill to the wall, mmap-drain (putpages bypass the vop_write ENOSPC gate at hammer2_vnops.c:854 — that gate exists only in the write(2) path), then round-robin zero-pwrite hammer:

• In-window acceptance, 4/4 runs: OV_OK i=43..63 phase=W rot=0 fsync=-1 errno=32 free=4194304 — pwrite() returns 65536 at the moment the backend is failing everything (console storm xop_strategy_write: error 32 loff=0000000031c0000..., hammer2_chain_create_indirect: error 00000020, backend unable to insert inode). fsync(2) leaks the raw HAMMER2 error code as errno 32 (EPIPE) — a separate mapping bug noted for the record.
• The lie: stage5 cached readback old=0 zeros=64 mixed=0 err=0 — every victim reads zeros; the overwrite "happened".
• The resurrection: after churnroot (740MB of root-fs reads to cycle the buffer cache; the wedged mount itself is not touched), stage7 readback old=1 zeros=57 mixed=6 err=0. v54 block1 hexdump (live mount): 44 46 32 36 34 32 2d 56 35 34 2d 42 31 2d 4f 4c 44 = "DF2642-V54-B1-OLD-" — the pre-overwrite pattern, byte for byte. v43: 53KB zeros + 11KB pattern (partial page recycling).
• Oscillation: the same offset subsequently read zeros again (stage9) and then the OLD pattern again (direct od minutes later) — repeated reads of one offset return different content, no error.
• Controls: v0-v42 (overwritten while backend healthy) stayed zeros through every churn — the churn does not fabricate staleness.
• Manifestation is racy (run1: 7/21 in-window victims flipped; runs 2-4: 0 flipped — delete COWs found small-radix freemap space): the delete failure depends on radix availability at the instant of the parent COW. The 4/4 constant is the silent in-window acceptance itself; run1 demonstrates the full stale-read consequence.

Offline media note: at the wall the flusher cannot rotate the volume header (DF-2633's post-crash observation), so walk2642.py finds the on-media topology pre-dating the victims; the decisive channel is the live-mount read after cache eviction (the chain cache + data-block buffers hold the pattern — exactly what the read path serves).

3. Exploit chain

Not a memory-corruption bug; the deliverable is integrity: an unprivileged-pattern local workload on a full hammer2 PFS silently fails to overwrite data with zeros and later serves the old content. "Overwrite with zeros" is the canonical secure-delete step — data an application believed erased comes back. Impact ceiling: silent data integrity violation + unkillable read inconsistency; no privesc.

4. Fix (fix.diff, validated)

-           hammer2_chain_delete(*parentp, chain,
-                        mtid, HAMMER2_DELETE_PERMANENT);
+           *errorp |= hammer2_chain_delete(*parentp, chain,
+                           mtid,
+                           HAMMER2_DELETE_PERMANENT);
+           if (*errorp == 0)
+               ++hammer2_iod_file_wzero;

With the error propagated, the strategy completion failure branch (strategy.c:675-681) runs for failed zero-deletes: the buffer is completed with B_ERROR/EIO, the console gains an explicit error line for the zero block, subsequent reads of the lbase return EIO, and the buffer-daemon retry keeps trying the write. Baseline vs patched transcripts: fix_validation.log (kernel #2). Recommended follow-up (not in this minimal fix): map the completion error through hammer2_error_to_errno() for bp->b_error, and fix the raw-code-32 fsync leak.

5. Honest limits

• The stale-read manifestation frequency is racy (freemap radix availability); the silent-acceptance half is 4/4 deterministic.
• The oscillation dynamics (chain-cache vs buffer-cache vs flusher at the wall) were observed but not fully reduced to a single ordering — both orderings were seen live.
• Guest wedges at the wall (umount EBUSY, sync hangs) are pre-existing DF-2633 behavior, used as-is; every attempt ended with a crash-reboot without snapshot revert.