Post-reset OBFL refill is a no-op: sense_bufs[].posted survives the controller soft reset, so tws_init_obfl_q() re-run by the recovery path posts nothing β firmware left with zero response frames, permanent I/O hang after any successful reset recovery
Summary
init_obfl_q only posts frames whose posted flag is false, but the flags are cleared only at attach and one overrun path - never on the reset path that must re-post every frame (the soft reset wiped the fw's free list). Recovery completes and every post-recovery command still times out forever (init_connect's own 60s poll runs through the queue it failed to refill) - a recoverable hiccup becomes a permanent adapter hang. Clear-flags-on-reset-path fix. Hardware-gated (no 9750 on guest).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4841 Β· 8 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 3.3 KB | β raw | |
| VERDICT.md | β | 3.2 KB | β raw | |
| trigger.c | β | 2.2 KB | view raw | |
| run.sh | β | 132 B | view raw | |
| guest_probe.log | β | 94 B | view raw | |
| env.txt | β | 641 B | view raw | |
| fix.diff | β | 1.9 KB | view raw | |
| verdict.json | β | 4.3 KB | view raw |
DF-4841 β post-reset OBFL refill is a no-op: sense_bufs[].posted survives the soft reset, leaving the firmware with zero response frames
sys/dev/raid/tws/tws_hdm.c:118-148 (tws_init_obfl_q) + the reset/recovery
path tws_cam.c:1198-1266.
What
sense_bufs[i].posted tracks "this response frame is currently in the
firmware's outbound free list". At attach, tws_init_reqs() clears all
flags (tws.c:620) and tws_init_ctlr() β tws_init_obfl_q()
(tws_hdm.c:111) posts all 256 frames.
The reset recovery path (tws_reset β tws_reset_cb β tws_reinit,
tws_cam.c:1198-1266) exists precisely because a soft reset reinitializes
the controller's message unit β the firmware's copy of the free list is
gone. tws_reinit() resets obfl_q_overrun, re-runs
tws_init_connect() and calls tws_init_obfl_q() (tws_cam.c:1262-1266)
to refill the queue. But posted is still true for every frame β
nothing on the reset path clears it (grep: cleared only at attach,
tws.c:620, and on an OBFL-overrun repost failure, tws_cam.c:543). The
"refill" loop
tws_hdm.c:127-144
while ( i < tws_queue_depth ) {
if ( !sc->sense_bufs[i].posted ) { /* never taken */
... post frame ...
sc->sense_bufs[i].posted = true;
posts nothing. The firmware is left with zero outbound free frames:
tws_init_connect()itself polls up to 60 s (TWS_POLL_TIMEOUT) for a response it can never receive β recovery burns a full minute then fails (and per DF-1926 also leaks the internal request slot).- Even if recovery "completes", every subsequent command submission cannot be answered β all I/O to the adapter times out permanently. The adapter and everything on it is dead until reboot.
One reset β triggered by any controller error that reaches tws_reset()
(bus reset, firmware fault, the unguarded re-entry of DF-4401, DF-4590's
callout storm) β converts a recoverable hiccup into a permanent hang of
the storage path.
Verification status
Not reproduced on the audit guest β hardware-gated. The audit guest
has no LSI/AVC 3ware 9750 PCI device, QEMU does not emulate one, and tws
is compiled into X86_64_GENERIC with nothing attached
(guest_probe.log: NO_3WARE_PCI, dev.tws absent). The defect is
certain from the driver's own code shape: the recovery path resets
obfl_q_overrun and re-runs the refill function intending to
re-post, while the state that makes the refill a no-op is left
untouched β the two cannot both be correct, and if the flags were
correct as-is (queue survives reset) the refill call would be pointless.
Fix
fix.diff β clear the posted flags on the reset path
(tws_reset_obfl_q() in tws_hdm.c, called from tws_reinit() before
tws_init_obfl_q()). Deliberately NOT done inside
tws_init_obfl_q() itself: its other caller (tws_submit_command() on
OBFL overrun, tws_hdm.c:282-284) must not double-post frames still
queued. git-apply clean; not runtime-testable (no controller).
Run (for a maintainer with hardware)
- Put the adapter into reset recovery (e.g.
camcontrol reseton the tws sim, or a controller fault). - Stock kernel: watch the recovery spin in
tws_init_connect's poll for TWS_POLL_TIMEOUT, then all I/O hang (iostatflatlines, every ccb times out). - Patched kernel: recovery completes and I/O resumes.
DF-4841 β VERDICT
Finding: post-reset OBFL refill no-op β sc->sense_bufs[i].posted
survives the controller soft reset, so tws_init_obfl_q() re-run by the
recovery path posts nothing and the firmware is left with zero response
frames β permanent I/O hang after any reset recovery.
Reproduced? NO β hardware-gated on this guest (defect certain from source)
guest_probe.log: no 3ware (13c1) PCI device on the guest, dev.tws
absent, tws not a loadable module here (compiled into X86_64_GENERIC).
QEMU cannot emulate a 9750 controller. Same verification posture as the
other tws hardware findings (DF-1924/1925/1926/4590/4591).
Why the defect is certain from the driver's own code
- The reset path re-runs the refill intending to re-post:
tws_reinit()resetssc->obfl_q_overrun = falseand callstws_init_obfl_q(sc)(tws_cam.c:1262-1266) β a function whose only job is putting frames into the firmware's outbound free list. - The refill posts only frames with
!posted(tws_hdm.c:128). postedis set true for all 256 frames during normal operation (tws_hdm.c:141) and is cleared at exactly two sites: attach (tws_init_reqs, tws.c:620) and a failed repost on OBFL overrun (tws_cam.c:543). Neither runs on the reset path (tws_reset β tws_reset_cb β tws_reinit, tws_cam.c:1198-1266).- Therefore, after a soft reset wipes the firmware's queue state,
tws_init_obfl_q()posts nothing. Invariants (2)+(3) and the intent (1) are mutually exclusive β the recovery path is self-defeating:tws_init_connect()polls 60 s (TWS_POLL_TIMEOUT, tws.h:74) for a response through the very queue it failed to refill, then every subsequent command times out forever. (If one instead assumed the queue SURVIVES the reset, theobfl_q_overrun=false+ refill call intws_reinitwould be dead code.)
Impact ceiling: local DoS of the storage adapter and all I/O bound to
it (system-wide stall for root/boot disks on it), reachable from any
controller error that enters tws_reset() β including DF-4401's
unguarded re-entry and DF-4590's stale-callot storms. Root/privileged
or hardware-fault trigger; Low severity per the audit rubric
(aligned with DF-1924/1925/1926).
Distinct from known findings: DF-1924/1925 (unbounded spins β hang inside the spin loops), DF-1926 (request-slot leak on init_connect failure). This finding is the successful-reset case: reset completes, recovery runs to completion, and the adapter is still dead because the refill was a no-op.
Fix
fix.diff: tws_reset_obfl_q() (tws_hdm.c) clears all posted flags;
tws_reinit() (tws_cam.c) calls it before tws_init_obfl_q().
Not inside tws_init_obfl_q() itself β the OBFL-overrun caller
(tws_hdm.c:282-284) must not double-post frames still queued.
git-apply clean against the tree (dry-run verified). Not runtime
validated: no controller on the audit guest.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testableFix not validated at runtime: no 3ware controller on the audit guest and QEMU cannot emulate one. fix.diff is git-apply clean (dry-run verified), places the clearing on the reset path only (not inside init_obfl_q, whose OBFL-overrun caller must not double-post frames still queued), and is validated by inspection against the four-step invariant argument in VERDICT.md.
findings/poc/DF-4841/fix.diff
Confirmed kernel references
Detail
Evidence (decisive lines)
VERDICT.md (four-step invariant argument with line receipts); trigger.c (hardware-gated trigger recipe); guest_probe.log (no 3ware PCI dev, dev.tws absent); fix.diff (tws_reset_obfl_q clearing posted flags on the reset path only - deliberately NOT inside init_obfl_q, whose OBFL-overrun caller must not double-post; git-apply dry-run clean)
PoC changes
No seed pack. Authored pack: trigger recipe + hardware-absence probe + fix.diff.
Verified recommended fix
Clear sense_bufs[].posted on the reset path before re-running tws_init_obfl_q() (tws_reset_obfl_q() called from tws_reinit; see fix.diff).
Verdict
tws_init_obfl_q() (tws_hdm.c:118-148) only posts response frames whose sense_bufs[i].posted is false; the flags are cleared at exactly two sites - attach (tws_init_reqs, tws.c:620) and an OBFL-overrun repost failure (tws_cam.c:543) - neither of which runs on the reset/recovery path (tws_reset -> tws_reset_cb -> tws_reinit, tws_cam.c:1198-1266). tws_reinit() nevertheless resets obfl_q_overrun and re-runs the refill (tws_cam.c:1262-1266), which is therefore a complete no-op after the soft reset wipes the firmware's copy of the outbound free list: the firmware is left with zero response frames, init_connect's own 60s poll (tws.h:74) cannot be answered, and every post-recovery command times out forever - a recoverable controller hiccup becomes a permanent adapter hang (local DoS of the storage path; root/hardware-fault trigger). The defect is certain from the driver's own invariant structure (the refill intent and the posted flags cannot both be correct), but hardware semantics of the soft reset were inferred, not measured - hence 'likely'. Not reproduced: the audit guest has no 3ware 13C1 PCI device, dev.tws is absent, and QEMU cannot emulate a 9750 (guest_probe.log); same posture as DF-1924/1925/1926/4590/4591. Distinct from DF-1924/1925 (hang inside spin loops) and DF-1926 (init_connect slot leak): this is the successful-reset case where recovery completes and the adapter is still dead because the refill posted nothing.
No comments yet.