DF-2633 / verdict.json
{ "finding_id": "DF-2633", "status": "reproduced", "reproduced": 1, "impact": "dos", "confidence": "certain", "verdict": "Reproduced standalone on the stock kernel with a recording writer on a dedicated 256MB hammer2: (1) 8.7-8.9MB of write()-SUCCESSFUL 64KB buffers permanently fail backend allocation (134-137 distinct logical offsets stuck in an endless 'xop_strategy_write: error 32' retry storm, 213 console lines/s unbounded, 325891+ lines measured); (2) fsync(2) returns 0 while its own data is failing (FILL_FSYNC rc=0 errno=0 at peak storm) - POSIX violation, the errno channel is entirely absent (strategy.c:670-681 sets b_error=EIO and prints; hammer2_vop_fsync ignores the vfsync return at vnops.c:240); (3) the wedged mount cannot be unmounted (EBUSY forever) and the only escape is a crash; (4) after the crash the recovered filesystem is COMPLETELY EMPTY - all 251,658,240 write()-successful, fsync()==0 bytes silently lost (postcrash_verify.log); file-creation flavor drops the insert error on the floor (hammer2_inode.c:1760-1763 'XXX return error somehow?'). fix.diff adds an atomic inode error field produced by the strategy-xop/chain-ins/chain-sync failure paths (rate-limited prints) and consumed by write(2) entry (user I/O) and fsync(2) exit; validated on the identical-config rebuilt kernel: FILL_FSYNC rc=-1 errno=28, APPEND_FSYNC rc=-1 errno=28, all retry writes/fsyncs -1 ENOSPC, console rate-limited to exactly 1 line/s, healthy-image dd+md5 roundtrip unaffected.", "exploit_chain": "", "evidence": [ "stock_run.attempt1.log - write() ENOSPC at the wall but FILL_FSYNC rc=0 errno=0 while the storm rages", "storm_measurements.txt - 325891+ console lines, 213/s, 134 distinct failing offsets (8.7MB of accepted-then-lost writes), 30s growth measurement", "umount_hang.attempt1.log - EBUSY forever; only escape is a crash", "planD.log + postcrash_verify.log - md5-verified crash-state image recovers COMPLETELY EMPTY: 251,658,240 write()-successful bytes lost", "control_roundtrip.log - healthy fs + sync + clean umount/remount roundtrips md5-identical (methodology control)", "fix_validation.log + fix_run_roundtrip.log - fixed kernel: fsync/write return ENOSPC at the wall; console 1 line/s; healthy roundtrip unaffected" ], "kernel_refs": [ "sys/vfs/hammer2/hammer2_vnops.c:854-864", "sys/vfs/hammer2/hammer2_vnops.c:999-1213", "sys/vfs/hammer2/hammer2_vnops.c:216-276", "sys/vfs/hammer2/hammer2_vnops.c:240", "sys/vfs/hammer2/hammer2_strategy.c:537-565", "sys/vfs/hammer2/hammer2_strategy.c:624-631", "sys/vfs/hammer2/hammer2_strategy.c:670-681", "sys/vfs/hammer2/hammer2_inode.c:1743-1767", "sys/vfs/hammer2/hammer2_inode.c:1760-1763", "sys/vfs/hammer2/hammer2_vfsops.c:2685", "sys/vfs/hammer2/hammer2_vfsops.c:3076-3127", "sys/vfs/hammer2/hammer2_strategy.c:1310-1356", "sys/vfs/hammer2/hammer2_inode.c:1324-1326" ], "poc_changes": "write2633.c/phase2.c built from scratch per the finding (per-write rc/errno counters, statfs polling, wall-regime probes: creates/append/tail/mmap + fsync rc at each stage, deterministic content + healthy-fs md5 controls). verify2633.c needed sys/stat.h added for fstat(2). Plan D added the crash-state media snapshot (dd of the tmpfs image while mounted+storming, md5-verified) because the tmpfs image cannot survive the crash otherwise.", "attempts": 4, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 5400, "guest_dirty": 0, "build_cmd": "cc -O2 -o write2633 write2633.c && cc -O2 -o phase2 phase2.c && cc -O2 -o verify2633 verify2633.c", "run_cmd": "sh run_planD.sh (stock: silent loss + storm + wedge) / sh validate_fix.sh (fixed: ENOSPC surfaces)", "code_hash": "768f7ffe5ddbb083ae6df915efbbad01cd844ba6b77b852f51b338b7cda377d2", "notes": "impact recorded as dos (wedge/flood/livelock) with silent-data-loss as the dominant harm. Backend matters: vn-over-tmpfs reproduces the storm (fast backend out-runs the 10ms-cached free_nominal pre-check, vfsops.c:3086-3107); vn-over-hammer2 fills slower and the pre-check catches the wall without a storm. Residual after fix: the dirty-buffer retry loop itself remains stock behavior (errors now reported, console now 1/s); a fully-quiescing retry redesign is out of minimal-fix scope. Also observed once: O_CREAT returning ENOSPC while leaving the file present on a full volume (create error-path gap adjacent to this finding).", "recommended_fix": "Add an atomic inode error state (int error on hammer2_inode) produced by the strategy-xop allocation-failure, inode-insert and inode-sync error paths (rate-limited kprintf) and consumed by hammer2_vop_write entry and hammer2_vop_fsync exit so write(2)/fsync(2) return ENOSPC/EIO for backend failures instead of silently losing data.", "fix_status": "fixed", "fix_kernel_uname": "DragonFly 6.5-DEVELOPMENT #2: Sat Aug 29 08:00:46 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "fix_baseline_reproduced": 1, "fix_patched_reproduced": 0, "fix_verdict": "A/B on identical KERNCONF: stock #0 - FILL_FSYNC rc=0 during a 325891-line error storm and 100% silent loss after the forced crash; fixed #2 - FILL_FSYNC rc=-1 errno=28, APPEND_FSYNC rc=-1 errno=28, all retry writes/fsyncs -1 ENOSPC, console rate-limited to 1 line/s, healthy-image writes + md5 roundtrip unaffected. The silent-success behavior is GONE; errors surface at the latest at the next write/fsync.", "fix_evidence": "fix_validation.log (fill + phase2 + RETRY probes), fix_run_roundtrip.log, fix_run.log, fix_build.log" } |