DF-2633 — hammer2 write() silently succeeds while backend allocation fails on full PFS

ENOSPC never reaches the writer: write(2) keeps returning success (and fsync(2) returns 0) while the backend flusher fails to allocate, floods the console, wedges the mount (EBUSY forever), and the "successfully written" data is silently lost.

Reproduce (stock kernel)

cc -O2 -o write2633 write2633.c
cc -O2 -o phase2 phase2.c
truncate -s 256M /tmp/h33.img
vnconfig -c vn1 /tmp/h33.img
newfs_hammer2 -L DATA /dev/vn1
mkdir -p /mnt/h33 && mount_hammer2 /dev/vn1@DATA /mnt/h33
mkdir -p ctrl
./write2633 /mnt/h33 ctrl 6000        # fill to the wall (~5 min)
./phase2 /mnt/h33 ctrl                # probe the wall state
# console: unbounded "xop_strategy_write: error 32 loff=..." flood
#         (325891+ lines, 213/s, never stops — see storm_measurements.txt)
# umount:  EBUSY forever (umount_hang.attempt1.log)

Observed (stock, planD.log / stock_run.attempt1.log):

• fill reaches the wall: write() ENOSPC at block 3836-3841 — but the last ~8-9 MB of write()-SUCCESSFUL 64KB buffers fail asynchronously: 134-137 distinct loffs stuck failing xop_strategy_write: error 32 (HAMMER2 NOSPC) forever.
• FILL_FSYNC rc=0 errno=0 — fsync(2) returns 0 while its own data is failing at the backend (POSIX violation), at 325k+ console errors.
• creates in the wall: 500/500 ENOSPC (pre-check catches them here).
• umount: Resource temporarily unavailable forever; sync hangs; the only escape from the wedged mount is a crash.
• crash + remount (media snapshot md5-verified while mounted+storming): the filesystem recovers COMPLETELY EMPTY — 251,658,240 bytes that write() returned success for and fsync() returned 0 for: 100% silent loss (postcrash_verify.log). Control: healthy fs + sync + clean umount/remount roundtrips md5-identical (control_roundtrip.log).

Fix validation (kernel #2 + fix.diff)

validate_fix.sh (+ fix_validation.log, fix_validation_roundtrip.log):

• at the wall: FILL_FSYNC rc=-1 errno=28, APPEND_FSYNC rc=-1 errno=28, subsequent writes AND fsyncs all -1 ENOSPC (RETRY_* lines) — the deferred error now surfaces through write(2)/fsync(2);
• console: rate-limited to exactly 1 line/s (stock: 213+/s unbounded);
• healthy image: dd 1.1GB/s, md5 roundtrip MATCH, umount clean.

Files

write2633.c / phase2.c / verify2633.c      harness
run_df2633.sh / run_planD.sh / validate_fix.sh
stock_run.attempt1.log / planD.log / phase2_run2.log
storm_measurements.txt     (console-storm + loss quantification)
umount_hang.attempt1.log   (EBUSY-forever transcript)
postcrash_verify.log       (recovered fs completely empty)
control_roundtrip.log / control_healthy.log
fix_validation.log / fix_validation_roundtrip.log
fix.diff (shared DF-2632+DF-2633), VERDICT.md, verdict.json, manifest.json

DF-2633 — VERDICT

Finding: hammer2 write(2) silently succeeds while backend allocation fails on a full PFS — ENOSPC never reaches the writer; fsync(2) returns 0 over failing data; the loss is silent and (in the observed regime) total.

Classification: REPRODUCED (silent data loss + fsync lies + console flood + permanent mount wedge), fix FIXED (errors surface, flood rate-limited; mount-wedge/livelock residual documented).

1. Where the error is lost (source trace)

• write(2) → hammer2_vop_write (hammer2_vnops.c:830) → hammer2_write_file (:999-1213) writes into the logical buffer cache (getblk/uiomovebp/cluster_write) and returns success. The only space check is the frontend pre-check hammer2_vfs_enospace(ip, uio_resid, cred) (:854-864) whose free_nominal is cached per tick (10 ms, hammer2_vfsops.c:3086-3107: pmp->free_ticks != ticks) — so a fast writer out-runs the cache and buffers are accepted after the allocator is actually empty.
• The real allocation happens later, asynchronously: hammer2_strategy_write (hammer2_strategy.c:537-565) → xop → hammer2_write_file_core (:624-626) → hammer2_assign_physical (:717+) → hammer2_chain_create → freemap. On a full PFS this fails with HAMMER2 NOSPC (0x20); the error handler (strategy.c:670-681) does: unbounded kprintf per failing buffer (the 213/s console flood), bp->b_error = EIO, biodone — and nothing else. write(2) already returned; no inode-level error state exists to carry the failure to the next syscall.
• fsync(2) → hammer2_vop_fsync (hammer2_vnops.c:216-276): vfsync(...) return value is ignored (:240), bio_track_wait waits for I/O completion (errors included) but discards them; hammer2_inode_chain_sync/chain_flush only see chain-level errors, never buffer-level ones → fsync returns 0 while its own buffers failed (measured: FILL_FSYNC rc=0 at 325,891 console errors).
• File-creation flavor: the syncer inserts new inodes via hammer2_inode_chain_ins (hammer2_inode.c:1743-1767) whose error path is kprintf(...); /* XXX return error somehow? */ (:1760-1763), called from the syncer with the return ignored (hammer2_vfsops.c:2685) — files whose inode never made it to media appear created (dirent live) and vanish on remount (DF-2628's dd observations).

2. Measured stock behavior (this run)

Single-filler 256MB hammer2 (vn over tmpfs — reproduces the fast-backend conditions; vn-over-hammer2-root behaves slower and the pre-check catches the wall first, see §4):

• write() ENOSPC at block 3836-3841 (251.4-251.7MB); free floor 5568KB.
• 8.7-8.9 MB of write()-successful 64KB buffers permanently failed at the backend: 134-137 distinct logical offsets stuck in xop_strategy_write: error 32 retry forever (storm_measurements.txt).
• FILL_FSYNC rc=0 errno=0 while the storm is active — POSIX violation.
• Console flood: 325,891+ lines, growth 213 lines/s, never terminates; guest degrades (ps ax >120 s).
• umount: EBUSY forever (dirty buffers can never flush) — the only escape from the wedged mount is a crash (umount_hang.attempt1.log).
• Crash + recovery: media snapshot taken while mounted+storming (crash semantics, md5-verified) recovers as a completely empty filesystem — all 251,658,240 write()-successful bytes gone (postcrash_verify.log: df 0 used, no fill.bin). No error was ever delivered to the writer.
• Controls: healthy fs + sync + clean umount/remount → md5 MATCH (control_roundtrip.log); controls written to the root fs survived crashes (control_healthy.log).
• Wall-state probes (phase2): creates 500/500 ENOSPC, append write ENOSPC, APPEND_FSYNC rc=0 (again the lie), tail.bin/mm.bin O_CREAT ENOSPC. Note: on DragonFly the same O_CREAT failure can still leave the file present (observed once in the DF-2632 natural-name probe on a space-exhausted volume: create returns ENOSPC, file exists afterward) — consistent with the error-handling gaps traced above.

3. Fix (fix.diff, validated)

Minimal correct channel modeled on UFS softdep-style deferred errors:

• struct hammer2_inode gains an atomic int error (errno) field (hammer2.h).
• Producers:
• hammer2_xop_strategy_write failure branch (strategy.c:670-681): atomic_swap_int(&ip->error, hammer2_error_to_errno(error)), bp->b_error now the mapped errno (ENOSPC, was EIO), and the console print becomes krateprintf (1/s).
• hammer2_inode_chain_ins error path (inode.c:1760-1763): records ip->error, rate-limited print (was the unbounded "backend unable to insert inode" flood).
• hammer2_inode_chain_sync error path (inode.c:1725-1733): same.
• Consumers:
• hammer2_vop_write (vnops.c:830+): at entry (user I/O only; the VM pager's UIO_NOCOPY writes are excluded) reports and clears the deferred error — the next write(2) after an async failure returns ENOSPC instead of silently continuing.
• hammer2_vop_fsync (vnops.c:216-276): after vfsync + chain flushes, reports and clears ip->error — fsync(2) now returns ENOSPC for failed backend writes.

4. Fix validation (kernel #2 = stock config + fix.diff)

A/B, same 256MB vn-over-tmpfs fill:

fix_validation.log, fix_validation_roundtrip.log.

5. Residual / honest limits

• The dirty-buffer retry loop itself is stock behavior and remains: in the exhausted state the failed buffers keep being retried (now silently except 1/s), the mount stays busy, and the guest is heavily loaded — a soft livelock until space is freed or the fs is unmounted by force. My fix does not change the retry design (out of minimal-fix scope); it guarantees the writer is TOLD (write/fsync ENOSPC) and the console no longer floods. fix_status: fixed for the reported finding (error surfacing + flood); the livelock is noted as residual.
• The pre-check's per-tick staleness (vfsops.c:3086-3107) still lets a fast writer accept a final buffer batch; those now surface as ENOSPC on the next write/fsync instead of being lost silently — the POSIX contract is restored.
• 100%-empty recovery after crash: dominated by the retry-wedge (no transaction could ever commit). With errors now reported, a writer that checks its return codes stops feeding the wedge; the crash-loss ceiling collapses to the accepted-but-unreported window, which the fix reports.