DF-3844 — blocked-cmd timeout releases mfi commands while the firmware still owns them

mrsas_issue_blocked_cmd() (mrsas.c:2865) and mrsas_issue_blocked_abort_cmd() (mrsas.c:3465) time out after 180 s and simply return; every caller then hands the command back to the free pool (mrsas_release_mfi_cmd, mrsas.c:2571) although the firmware still holds the SMID. The late completion is routed onto the recycled command via sync_cmd_idx (mrsas.c:1294-1299 → 2913), waking the wrong waiter with a stale status, and — when the recycler is a RAID-map sync — double-releasing the command into mrsas_mfi_cmd_list_head (mrsas.c:2953/2960), duplicating it on the free list so two threads later obtain the same mfi command (endpoint: double completion / double xpt_done).

Most user-reachable entry: mrsas_passthru ioctl ignores the blocked-cmd return (mrsas_ioctl.c:301) and releases unconditionally (mrsas_ioctl.c:367); the device node is 0660 root:operator (mrsas.c:790-792).

Reproduce (needs hardware)

See df3844_trigger.c (sketch for equipped machines) and VERDICT.md for the full source-level interleaving. On the audit guest this cannot run: mrsas is compiled into the kernel (kldstat -v → 80 pci/mrsas) but no LSI/Avago PCI device exists, so /dev/mrsas0 is absent (run.log).

Files

• VERDICT.md — full narrative + interleaving
• df3844_trigger.c — trigger sketch for equipped hardware
• run.log — guest module/hardware presence check
• env.txt — guest environment
• fix.diff — timeout abort + per-command wait channel + ioctl return check
• manifest.json, verdict.json

VERDICT — DF-3844 (blocked-cmd timeout vs completion: mfi cmd released while firmware-owned)

status: inconclusive (hardware-gated; source-certain defect, timing-dependent manifestation) confidence: likely · class: lifetime/serialization defect with pool-corruption endpoint

The defect (source, certain)

mrsas_issue_blocked_cmd() — sys/dev/raid/mrsas/mrsas.c:2865-2900: after total_time >= max_wait (180 s, MRSAS_INTERNAL_CMD_WAIT_TIME, mrsas.h:1841) it breaks with retcode = 1 and returns without aborting the command at the firmware and without marking it abandoned. Every caller then releases the command back to the free pool while the firmware still owns the outstanding SMID:

• mrsas_get_seq_num — mrsas.c:566-573 (return ignored; memcpy + release)
• mrsas_shutdown_ctlr — mrsas.c:3058-3059
• mrsas_flush_cache — mrsas.c:3098-3099
• mrsas_passthru (ioctl) — mrsas_ioctl.c:301 ignores the return and mrsas_ioctl.c:367 releases unconditionally ← reachable by group operator (node is 0660 root:operator, mrsas.c:790-792)
• mrsas_issue_blocked_abort_cmd — mrsas.c:3493-3516 releases the abort command on timeout the same way, and leaves the victim (sc->aen_cmd, sc->map_update_cmd) outstanding with no retry (AEN channel wedges: mrsas_aen_handler returns early at mrsas.c:3608 while sc->aen_cmd != NULL).

Consequence chain (interleaving)

1. Thread A's passthru times out; mrsas_release_mfi_cmd(cmd_A) puts cmd_A on mrsas_mfi_cmd_list_head (mrsas.c:2571-2582). The mpt passthru wrapper for A (sync_cmd_idx = A->index, mrsas.c:2821) is still outstanding at the firmware.
2. The firmware completes A late → ISR → mrsas_complete_cmd PASSTHRU case (mrsas.c:1294-1299) → cmd_mfi = sc->mfi_cmd_list[cmd_mpt->sync_cmd_idx] = cmd_A even though cmd_A is free or already re-issued to thread B → mrsas_complete_mptmfi_passthru(sc, cmd_A, …) (mrsas.c:2913).
3. If cmd_A was recycled by B: B's blocked wait is woken with A's stale status (mrsas_wakeup, mrsas.c:3001-3014, writes cmd->cmd_status on a command owned by another thread). If B's command happens to be a map sync (MR_DCMD_LD_MAP_GET_INFO, mbox.b[1]==1), the completion path releases cmd_A at mrsas.c:2953/2960 while B still owns it; B's own completion then releases it again → the same cmd inserted twice into mrsas_mfi_cmd_list_head → two future mrsas_get_mfi_cmd() callers receive the same command → concurrent frame use and double completion (double xpt_done on a shared ccb is the eventual memory-safety endpoint, via mrsas_cmd_done, mrsas_cam.c:1080-1089).
4. Independently, mrsas_reset_ctrl's re-fire loop (mrsas.c:2448-2470) re-fires every mpt cmd with sync_cmd_idx != ULONG_MAX after OCR — including wrappers whose mfi cmd was already released by a timed-out waiter (no ownership check), feeding step 3 post-reset.
5. The wait/wake protocol uses one shared ident: sc->chan = (void*)&cmd; tsleep((void *)&sc->chan, …) (mrsas.c:2880-2888, 3011-3012) — wakeup_one may wake the wrong waiter; the completed command's thread can then sleep to full timeout, extending window 1-3.

Why not reproduced on the guest

Requires a MegaRAID HBA (driver attaches, /dev/mrsas0 exists) and firmware that stalls or loses a passthrough completion for >180 s while a second ioctl recycles the command. The guest has mrsas compiled in (kldstat -v → 80 pci/mrsas) but no LSI PCI device and no /dev/mrsas* node (run.log). Timing-gated even on equipped hardware. Hence inconclusive, likely confidence (defect structure is source-certain; manifestation is probabilistic).

Fix

fix.diff: (a) abort the command at the firmware on timeout before returning (mirrors Linux megaraid_sas megasas_internal_reset-style discipline), (b) make the wait channel per-command (tsleep(cmd) / wakeup_one(cmd)) instead of the shared sc->chan ident.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.