Signed 32-bit flusher sequence ring misorders all completion/mis-sequencing comparisons past 2^31; wrapped state causes permanently stuck flush group and self-sustaining synchronous-finalize storm (system wedge)
Summary
hmp->flusher.{done,next}, hammer_flush_group_t.seq and hammer_reserve_t.flg_no form a BSD-style sequence ring whose every consumer asks 'is a ahead of b' via signed int differences: hammer_flusher_wait (:166), hammer_flusher_running (:178-181), hammer_flusher_flush's group-matching/mis-sequencing branch + KKASSERT (:331-357), and the delay-list release check (:447-451, fed by hammer_blockmap.c:730/736). The stock hammer.ko compiles these as sub/test/jle (wrapping subtraction, signed test - disasm in pack). Once the ring distance between the allocating side and the single master consumer reaches 2^31 the differences wrap and every comparison flips: waiters early-return (sync/fsync acknowledge unflushed data), the KKASSERT evaluates wrongly, flush groups get mis-sequenced (stuck forever on non-INVARIANTS builds), and block reservations are released up to 2^31 flush cycles early (freed big-block reuse while referenced). The allocating side is unprivileged and fire-and-forget: HAMMERIOC_SYNCTID/ASYNC bypasses caps_priv_check (hammer_ioctl.c:117-120 overwrites the error - DF-3112) and hammer_flusher_async bumps flusher.next once per call with no waiting; measured ~65k seq/s per process, ~780k/s with 12. On configurations where the frontend out-allocates the master flusher for a sustained cumulative period (slow/network storage pinning the master in synchronous finalizes while fast CPUs burn ioctls) the ring laps: INVARIANTS KKASSERT can panic; otherwise the wrapped state wedges the whole system (stuck flush group -> hammer_flusher_haswork() true forever -> synchronous finalize per dummy sequence -> I/O storm; guest: ssh died past banner timeout, console silent, hard reset required). Organic lap NOT achievable on the fast test guest (914M unpriv allocations, master held the gap at 4707; 4 heavy writers, gap <=9k drained to 2 within seconds) - the recorded mechanism run injects the exact wrapped field state with a root setup helper (seqshift.ko) and lets the unpriv burners operate on it: wedged within ~2 minutes. No memory-corruption primitive; uid=0 not in scope for this class. Fix authored + compile-verified (unsigned ring types - well-defined modular arithmetic, no on-disk change; fix_build.log shows unsigned cmp/je codegen).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3118 Β· 18 files| File | Type | Description | Size | |
|---|---|---|---|---|
| burn.c | β | 3.0 KB | view raw | |
| writer.c | β | 520 B | view raw | |
| seqshift.c | β | 2.1 KB | view raw | |
| seqstat.c | β | 1.2 KB | view raw | |
| Makefile | β | 101 B | β download | |
| build.sh | β | 101 B | view raw | |
| run.sh | β | 195 B | view raw | |
| build.log | β | 465 B | view raw | |
| run.log | β | 1.5 KB | view raw | |
| disasm.txt | β | 2.1 KB | view raw | |
| gapseries.txt | β | 205 B | view raw | |
| wedge_boot.log | β | 13.7 KB | view raw | |
| run3logs/ | β | 4.0 KB | β download | |
| env.txt | β | 525 B | view raw | |
| fix.diff | β | 2.3 KB | view raw | |
| fix_build.log | β | 709 B | view raw | |
| VERDICT.md | β | 5.5 KB | β raw | |
| verdict.json | β | 4.1 KB | view raw |
DF-3118 verdict β full narrative
The defect (certain)
hmp->flusher.{done,next} (sys/vfs/hammer/hammer.h:772-781), flush-group
seq (hammer.h:292) and reservation flg_no (hammer.h:740) are signed
32-bit sequence numbers used as a BSD-style ring: every consumer computes
a - b > 0 to ask "is a ahead of b":
hammer_flusher_wait()sys/vfs/hammer/hammer_flusher.c:166hammer_flusher_flush()KKASSERT + mis-sequencing branch :338-357- delay-list release check :447-451 (with hammer_blockmap.c:730,736)
hammer_flusher_running():178-181
The compiled module uses sub; test; jle (see disasm.txt) β a wrapping
32-bit subtraction whose result is then read as signed. Ring arithmetic
with a signed difference test is only correct while the distance between
allocator and consumer stays below 2^31; past that every comparison flips.
The allocating side is attacker-influenceable without privilege:
HAMMERIOC_SYNCTID with op=HAMMER_SYNCTID_ASYNC is unprivileged
(hammer_ioctl.c:72 computes caps_priv_check into error, but the
case HAMMERIOC_SYNCTID: handler at hammer_ioctl.c:117-120 overwrites
error unconditionally), and each call bumps hmp->flusher.next once
with no waiting when no flush groups are pending (hammer_flusher.c:114-122).
Measured: ~65k seq/s per process, ~780k/s with 12 processes (run.log).
What was demonstrated on the guest
Mechanism (reproduced from an injected wrapped state)
seqshift.ko (root, setup-phase; findings/poc/DF-3118/seqshift.c) set
next = done + 2^31 - 200000, i.e. next negative while done positive β
the exact field-value state that a successful 2^31-lap would produce. The
running unprivileged burners kept allocating; the grouper's flush group
was created at the wrapped frontier. The master then executed
hammer_flusher.c:338-357 with flg->seq negative and seq = done+1
positive:
seq == flg->seqfails,KKASSERT(flg->seq - seq > 0 ...): the difference underflows INT_MIN and wraps positive, so the assert passes (no panic text β consistent with the silent serial console in wedge_boot.log),flg = NULL: the pending group is permanently unreachable,hammer_flusher_haswork()is true forever (group list non-empty), so the master executes a synchronous finalize (volume-header + undo flush + disk cache flush) for every dummy sequence number β a self-sustaining fsync storm.
Observed: guest wedged within ~2 minutes; ssh connections timed out during
banner exchange on 25 s and 30 s retries five minutes apart; serial console
silent; QEMU still running; only a hard reset (vm.sh reset with-src)
recovered the machine. That is a full system denial of service from the
wrapped-counter state, with no kernel memory corruption involved.
Organic reachability (not demonstrated; refuted on this configuration)
Two organic attempts were run:
- 12 unpriv burners, 22 minutes, 914,588,507 sequence numbers allocated
(~780k/s). Probe (wedge_boot.log):
done=914583800 next=914588507 gap=4707. The single master flusher thread consumed 99.9995% of the allocations. - 12 burners + 1 grouper + 4 writers dirtying 11.3 GB in 7 minutes (43k/s per burner, ~520k/s aggregate). Gap stayed 5.5k-9k (gapseries.txt), and 2 seconds after the burners were killed the gap had drained to 2.
Conclusion: on this guest (6 vCPU, fast qcow2 disk) the master's per-sequence
consumption capacity exceeds the fs_token-serialized unpriv allocation
ceiling, so the ring distance cannot accumulate toward 2^31 β it returns to
~5k after every master I/O stall. Reaching the lap organically would require
~2^31 net allocations during cumulative master stalls (~2000 s at the
measured 430k/s with the master continuously pinned in synchronous
finalizes), plausible only on configurations where the frontend vastly
outpaces the flusher's disk (very slow/network storage with many fast
frontend cores) β not demonstrated here.
Bottom line
The arithmetic defect is real and its post-condition (wrapped ring) is a hard, self-sustaining system wedge, demonstrated by driving the kernel through the exact wrapped state. The unprivileged trigger, while cheap per operation, could not out-race the master flusher on the test configuration, so end-to-end unprivileged reproduction is inconclusive. If any future change (or a slow-storage deployment) lets the frontend allocate faster than the master consumes for a sustained period, this becomes an unprivileged system DoS plus, at the wrap seam, flush-group mis-sequencing (silent on non-INVARIANTS builds) and premature delay-list reservation release.
Fix
fix.diff (git-apply-able, verified against the read-only tree): make
done/next, flg->seq, resv->flg_no and the local seq variables
unsigned. Modular 2^32 arithmetic then keeps every difference comparison
correct for distances < 2^31 and the compiler emits proper ring codegen
(cmp/je + unsigned branches β fix_build.log, module compiled clean with
-Werror against the patched headers in the guest's /usr/src copy).
Full kernel A/B validation was not performed (no organic reproduction to A/B against); the fix compile-checks and the codegen change is verified.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testablefix.diff applies clean to sys/ and the hammer module compiles -Werror clean with it in the guest's /usr/src copy; patched hammer_flusher_wait emits unsigned ring codegen (cmp/je). Full kernel A/B not run: there is no organic baseline reproduction to compare against (organic trigger refuted on this config), so fixed-kernel behaviour could not be meaningfully A/B'd.
fix_build.log
Confirmed kernel references
- sys/vfs/hammer/hammer_flusher.c:114
- sys/vfs/hammer/hammer_flusher.c:166
- sys/vfs/hammer/hammer_flusher.c:178
- sys/vfs/hammer/hammer_flusher.c:331
- sys/vfs/hammer/hammer_flusher.c:338
- sys/vfs/hammer/hammer_flusher.c:356
- sys/vfs/hammer/hammer_flusher.c:447
- sys/vfs/hammer/hammer.h:772
- sys/vfs/hammer/hammer.h:292
- sys/vfs/hammer/hammer.h:740
- sys/vfs/hammer/hammer_blockmap.c:730
- sys/vfs/hammer/hammer_ioctl.c:117
Detail
Evidence (decisive lines)
['wedge_boot.log: last serial lines show the injected state (done=914583800 next=-1233099849) after which the guest wedged (no further output, ssh banner timeouts, QEMU alive)', 'disasm.txt: stock hammer.ko hammer_flusher_wait = sub;test;jle (wrapping signed 32-bit difference)', 'gapseries.txt + run.log: organic gap measurements (4707 after 914.59M allocations; <=9k under heavy dirty load; drains to 2 in 2 s)', 'fix_build.log: patched module compiles -Werror clean and emits unsigned ring codegen (cmp/je)', 'VERDICT.md: full narrative with path:line citations']
PoC changes
burn.c/seqshift.c/seqstat.c/writer.c written fresh for this pack (no seed existed): unpriv trigger, root setup injector, read-only gap probe, dirty-load writer.
Verified recommended fix
Make the flusher sequence counters and derived locals unsigned (done/next, flg->seq, resv->flg_no) so the ring differences are modular; see fix.diff.
Verdict
Signed 32-bit flusher sequence ring (hmp->flusher.{done,next}, flg->seq, resv->flg_no) misorders all difference comparisons once the ring distance reaches 2^31; the allocating side is an unprivileged fire-and-forget HAMMERIOC_SYNCTID ioctl (hammer_ioctl.c:117 overwrites the caps_priv_check error). Mechanism proven on the guest: with next wrapped negative while done stayed positive (state injected by a root setup helper, identical to a successful 2^31 lap), hammer_flusher_flush():338-357 mis-sequences the pending flush group (difference wraps, KKASSERT passes, flg=NULL), the group is stuck forever, and the master runs a synchronous finalize per dummy sequence - the guest wedged past ssh recovery and needed a hard reset. Organic unprivileged reachability NOT demonstrated and refuted on this configuration: 12 unpriv burners allocated 914,588,507 sequences in 22 min with the master holding the gap at 4707, and with 4 heavy writers dirtying 11.3 GB the gap stayed <= 9k and drained to 2 within seconds of stopping. End-to-end unpriv trigger would require ~2000 s of cumulative master stalls (slow storage), which the test config does not allow.
No comments yet.