DF-2628 / attempts_enospace_race.log
DF-2628 — ENOSPC exhaustion-race attempts (guest, stock kernel #0 boot of this
verify run; hz=100 tick=10ms; 128MB hammer2 on /dev/vn1 backed by a tmpfs
image; free_reserved = allocator_size/20 ~= 2.8MB usable, root cut at /2)
[1] dd-based drain (8MB files, urandom data):
FILL: 33652 files x 8MB, elapsed 235s
/dev/vn1@DATA 221184 212992 8192 96%
-> writes blocked by the pre-check at ~4MB shown free; each subsequent
dd still exits 0 (creates an empty file), so the loop burns 33652
creates. Console during a later harness in the same state:
hammer2: backend unable to insert inode 0xfffff8011ee814c0 34667
hammer2_chain_create_indirect: error 00000020 No Space on Device
(x thousands, inode numbers incrementing)
=> allocation failures surface at the backend but write() itself
never fails synchronously; the "No Space" regime coexists with
df showing 4MB free (allocator accounting vs freemap bitmaps
diverge in this regime).
[2] C drain harness (drainfill.c, 256KB incompressible xorshift blocks into
a single file, ~660-950MB/s buffer throughput) + linkloop.c (kHz
link/unlink/stat loop), 20-35s windows:
--- t=10s df: 16768 used 204416 avail drain: drained 3934 MB (5s) HITs: 0
--- t=15s df: 29952 used 191232 avail drain: drained 7191 MB (10s) HITs: 0
--- t=20s df: 42880 used 178304 avail drain: drained 10384 MB (15s) HITs: 0
--- t=25s df: 55424 used 165760 avail drain: drained 13512 MB (20s) HITs: 0
--- t=30s df: 68096 used 153088 avail drain: drained 16641 MB (25s) HITs: 0
--- t=35s df: 80256 used 140928 avail drain: drained 19672 MB (30s) HITs: 0
-> allocator consumption (df "used") grows at ~1.3-2 MB/s while ~660MB/s
flows through the buffer cache; the file grew to 27.5GB on a 128MB
volume (async allocation failures silently absorbed). The in-tick
drain needed to open the stale-cache window is >=2.8MB/10ms =
280MB/s of ALLOCATOR consumption - ~150-300x beyond measured.
[3] zero-filled drain attempt:
"drained 14222 MB (15s)" with df showing 128 blocks used - hammer2 does
not allocate for all-zero data (zero-fill optimization); earlier
repetitive 0xA5/counter patterns were lz4-compressed to ~9.5MB.
-> drain data must be incompressible (fixed in drainfill.c), but even
then consumption stays flush-bound (see [2]).
[4] backing-device fault injection (all failed to surface a synchronous
error in hammer2_dirent_create):
a) tmpfs filled to 100% (0 blocks avail, 1KB-granular top-up) behind a
SPARSE vn image: link() x5 all SUCCEEDED (vn allocates image pages
via the VM object, bypassing tmpfs size accounting).
b) truncate -s 4M of the backing image mid-mount: link() x5 SUCCEEDED
(object grows past EOF silently).
c) vnconfig -u (detach) with hammer2 still mounted: link() SUCCEEDED
(mount's open reference keeps the vn alive).
d) vn image on a UFS volume filled to 109% (avail -15472): link() x5
SUCCEEDED (buffer-cache absorbs the device-write failures
asynchronously).
e) an earlier detached-mount state also left the console flooding with
"xop_strategy_write: error 32" at 3.4GB offsets (the async backlog
of a 27GB drainfile on a 128MB volume) - again asynchronous, never
visible to the link() caller.
CONCLUSION: on this guest/kernel there is no constructible vector that
delivers a synchronous hammer2_dirent_create() failure to an unprivileged
link(2). The deterministic window-exhaustion trigger (VERDICT.md 2) is
source-verified but the fill panics the stock kernel first (VERDICT.md 3.2).