DF-3845 — OCR return-to-CAM flush vs concurrent ISR completion (no serialization)

mrsas_reset_ctrl()'s return-to-CAM loop (mrsas.c:2335-2343) and the ISR completion path (mrsas_isr → mrsas_complete_cmd, mrsas.c:1200/1236, lock-free) both test mpt_cmd->ccb_ptr non-atomically and both call mrsas_cmd_done() for the same command when an in-flight interrupt races the reset. The only fence is a 1-second DELAY heuristic (mrsas.c:2327) — mrsas_disable_intr does not stop an already-delivered MSI on another CPU. Result: double xpt_done (CAM ccb UAF), possible xpt_done(NULL) panic, and double mrsas_release_mpt_cmd (duplicate free-list entry → two IOs sharing one mpt cmd). mrsas_scsiio_timeout (mrsas_cam.c:370) unconditionally re-arms a 600 s callout that callout_stop in mrsas_cmd_done cannot drain, so the stale timer fires on the command's next owner and re-triggers OCR.

Reproduce

Race needs an attached MegaRAID HBA (completions in flight + induced fault/timeout to start OCR). The audit guest has mrsas compiled in (kldstat -v → 80 pci/mrsas) but no LSI PCI device and no /dev/mrsas* (run.log) — the code path cannot be entered.

Files

• VERDICT.md — full interleaving + aggravator analysis
• run.log, env.txt — guest presence checks
• fix.diff — idempotent completion (atomic cmpset claim), no 600 s re-arm, reset-loop claim
• manifest.json, verdict.json

VERDICT — DF-3845 (OCR return-to-CAM flush vs concurrent ISR completion; callout re-arm on recycled cmd)

status: inconclusive (hardware-gated; race structure source-certain, overlap window timing-dependent) confidence: likely · class: missing serialization between reset path and completion path (DF-3813 family)

The race (source)

Sides that complete the same mrsas_mpt_cmd with no common lock:

Side A — ISR: mrsas_isr (INTR_MPSAFE, mrsas.c:1200-1223) → mrsas_complete_cmd (mrsas.c:1236-1344) runs with no lock held; for LD/Fast-Path completions it calls mrsas_map_mpt_cmd_status (writes cmd->ccb_ptr->ccb_h.status, mrsas.c:1353-1391) and then mrsas_cmd_done (mrsas_cam.c:1080-1089): callout_stop, mrsas_unmap_request, xpt_done(ccb), ccb_ptr = NULL, mrsas_release_mpt_cmd (TAILQ insert, mrsas_cam.c:605-615). Its only reset guard is test_bit(MRSAS_FUSION_IN_RESET, &sc->reset_flags) at mrsas.c:1213 — checked before the completion walk, with no memory barrier against the reset thread.

Side B — OCR thread: mrsas_ocr_thread (mrsas.c:2232-2273, holds recursive sim_lock) → mrsas_reset_ctrl (mrsas.c:2309-2498): 1. set_bit(MRSAS_FUSION_IN_RESET) at mrsas.c:2324, 2. mrsas_disable_intr at mrsas.c:2326 — but an MSI/line interrupt already delivered to another CPU still runs its handler, 3. DELAY(1000*1000) at mrsas.c:2327 — a 1-second heuristic that is the only fence, 4. mrsas_wait_for_outstanding (mrsas.c:2522) and, if commands remain, the return-to-CAM loop at mrsas.c:2335-2343: if (mpt_cmd->ccb_ptr) { … mrsas_cmd_done(sc, mpt_cmd); … }.

If an ISR entered mrsas_complete_cmd before step 1 and is still executing when the loop at step 4 runs (possible under a deep reply-queue drain with max_fw_cmds completions and slow xpt_done under heavy CAM load — the walk is unbounded until it meets an unused descriptor), both sides pass their non-atomic ccb_ptr check for the same cmd and both call mrsas_cmd_done:

• double xpt_done(ccb) — CAM completes a ccb twice → use-after-free in the CAM layer,
• xpt_done(cmd->ccb_ptr) racing cmd->ccb_ptr = NULL → possible xpt_done(NULL) panic,
• double mrsas_release_mpt_cmd → the cmd is inserted twice into mrsas_mpt_cmd_list_head (mrsas_cam.c:611) → two future IOs share one mpt cmd/frame.

Aggravator — timeout callout: mrsas_scsiio_timeout (mrsas_cam.c:352-375) re-arms itself for 600 s unconditionally after firing, and mrsas_cmd_done uses callout_stop (not callout_drain/callout_stop_sync, mrsas_cam.c:1082), so a callback already running on another CPU re-arms the callout on a command that completes and is recycled, firing 600 s later on an unrelated IO (cmd->ccb_ptr of the new owner is non-NULL) and setting do_timedout_reset → spurious OCR → the Side-A/Side-B race above, repeated on demand. This converts a rare coincidence into a recurring trigger once any IO has timed out once.

Why not reproduced on the guest

Needs a MegaRAID HBA generating completions plus an induced FW fault/IO timeout to start OCR. Guest has the driver compiled in but no LSI device and no /dev/mrsas* (run.log); CAM cannot even queue IOs to the SIM. Overlap probability is timing-dependent even on hardware. Hence inconclusive / likely.

Fix

fix.diff makes completion idempotent: mrsas_cmd_done claims the command via an atomic compare-and-set of ccb_ptr (first claimer wins; the loser returns without touching the ccb), and the reset loop uses the same claim. This closes the double-completion endpoint regardless of the delay heuristic; the diff also replaces the dummy 600 s re-arm in the timeout callback with a plain return when a reset is already flagged.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.