DF-4101 — Stale bttimeout fires BDR/hard-reset on a recycled CCB

Verdict: REPRODUCED (deterministic choreography of the exact race)

Rig class as DF-2429 (see DF-2429/README.md).

Build

./push_rig.sh

Run

sh run1.sh       # DF-4101 is the third section

Choreography (mirrors the production race window)

1. queue io ccb-A (healthy, 60 s timeout) → ACTIVE (flags=0x1).
2. hold — the ioctl thread takes bt->lock (the lock bttimeout acquires at bt.c:2143).
3. racefire 3 — a kthread calls the production bttimeout(bccb) and blocks acquiring the lock: this is the in-progress callout invocation that callout_stop at bt.c:1449 cannot cancel.
4. Under the lock: comp 3 completes ccb-A (BMBI_OK; the callout_stop inside btdone is a no-op against the blocked handler) and queue io ccb-B — production btgetccb hands out the same BCCB (bccb_kva identical), which btaction marks ACTIVE again with a fresh 60 s timer.
5. rel — the stale bttimeout acquires the lock and proceeds against ccb-B.

Expected output (run.log)

• After rel: ccb-B flags=0x7 (DEVICE_RESET set by the stale handler), timer=1 (stale handler's 2s re-arm), console CCB 0xfffff801167df620 - timed out — printed for a command whose own 60 s timer never expired. ccb-A meanwhile completed CAM_REQ_CMP (comp=1, last_status=0x1).
• 2 s later: the stale handler's re-arm fires the hard-reset branch — vhba_hard_resets increments (1→2; the extra hard reset on a healthy bus), and ccb-B completes ccb_status=0xb (CAM_CMD_TIMEOUT) — a healthy command misreported as timed out.

The lock choreography makes the production race window deterministic; every step except the timing is genuine production code.

DF-4101 VERDICT — stale bttimeout on a recycled CCB

Bottom line

REPRODUCED — the race mechanism, its preconditions and its full consequence chain, all on production code. The timing window (a bttimeout invocation entered but blocked on bt->lock) is exactly the window callout_stop (bt.c:1449) cannot close; the rig makes it deterministic by holding the lock while the stale handler runs, which is what genuine contention does.

Live trace (run.log + console)

[snap 3]  ccb-A: bccb=0xfffff801167df620 flags=0x1 timer=1   (healthy, 60s)
[rig] stale handler thread: calling production bttimeout(0xfffff801167df620),
      will block on bt->lock                                   <- in-progress callout
[comp 3]  ccb-A completed BMBI_OK (callout_stop a no-op against it)
[queue 4] ccb-B: bccb=0xfffff801167df620  <- SAME BCCB recycled by btgetccb
[rel]     lock released -> stale bttimeout proceeds on ccb-B
[snap 3]  ccb-A: comp=1 last_status=0x1 (CAM_REQ_CMP)
[snap 4]  ccb-B: flags=0x7 (DEVICE_RESET set by the STALE handler)
          timer=1 (stale handler's 2s re-arm)
          console: "CCB 0xfffff801167df620 - timed out"  (60s timer never fired)
[snap 4]  after 2s: ccb_status=0xb (CAM_CMD_TIMEOUT) — healthy command
          misreported; vhba_hard_resets 1 -> 2 (extra HARD_RESET(0x80)
          write on a healthy bus)

Every claim of the finding is visible: stale handler runs after the CCB was completed and freed; same BCCB recycled for a new command; stale handler sees BCCB_ACTIVE on the new command, freezes the SIM queue, arms BDR, and 2s later hard-resets a healthy bus; the new command completes CAM_CMD_TIMEOUT through no fault of its own.

Root cause (path:line)

• bt.c:1449 — callout_stop(&bccb->timer) does not interlock against an in-progress bttimeout (non-lock-aided callout; the handler is entered and blocked at bt.c:2143 lockmgr(&bt->lock, ...)).
• bt.c:2147 — the BCCB_ACTIVE check cannot distinguish the new command from the one that timed out (no generation handshake).

Impact

Under real contention (timeout firing while the completion path holds bt->lock), a healthy command is misreported CAM_CMD_TIMEOUT, the SIM queue is frozen, and the bus takes an unnecessary BDR followed by a hard reset 2s later — I/O disruption across all targets (DoS-class reliability bug, sili DF-4027 / amr DF-4060 family).

Rig fidelity note

The only synthetic element is the guarantee that the handler is mid-execution (the kthread stands in for the fired callout; in production that requires the lock to be held at fire time — precisely the race condition). Everything from bttimeout through btdone, btaction, btgetccb recycling and the hard reset is production code from the shipped kernel image, and the console output is the driver's own.

Fix validation

fix.diff implements the generation handshake: bt_ccb gains gen/timer_gen; btgetccb bumps gen, btexecuteccb snapshots it into timer_gen when arming, and bttimeout/btdone ignore invocations whose generation does not match. Patched bt.c+btreg.h compile clean with the kernel's own flags under -Werror (FIXED-COMPILES-CLEAN). No runtime A/B kernel build (race/DoS class, not memory corruption; not mandated).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.