DF-3892 — mpssas_scsiio_timeout runs lock-free (callout_init_mp) on state owned by mps_lock: guaranteed KKASSERT panic on the first SCSI IO timeout (INVARIANTS/GENERIC), unsynchronized race on stripped kernels

File: sys/dev/raid/mps/mps_sas.c (pass 2)

What

struct mps_command.cm_callout is initialized with callout_init_mp() (sys/dev/raid/mps/mps.c:860) — DragonFly's lock-free MP-safe callout (sys/kern/kern_timeout.c:767-771 sets CALLOUT_MPSAFE, and softclock then executes the callback with no lock held, kern_timeout.c:626-631; only callout_init_lk()/CALLOUT_AUTOLOCK holds a lock across the callback, kern_timeout.c:609-625).

But the callback armed on that callout for every SCSI IO (callout_reset(&cm->cm_callout, ..., mpssas_scsiio_timeout, cm) at sys/dev/raid/mps/mps_sas.c:1768) is written as if it ran under sc->mps_lock:

static void
mpssas_scsiio_timeout(void *data)            /* mps_sas.c:1481 */
{
        ...
        cm = (struct mps_command *)data;
        sc = cm->cm_sc;

        KKASSERT(lockowned(&sc->mps_lock));   /* mps_sas.c:1490 — FALSE here */
        ...
        mps_intr_locked(sc);                  /* mps_sas.c:1499 — lock-free! */

Every field it then touches (cm->cm_state, cm->cm_ccb, targ->timedout_commands, targ->tm) is otherwise exclusively protected by mps_lock, and mps_intr_locked() itself is a lock-held-context-only function (all its callers wrap it in mps_lock/mps_unlock, e.g. sys/dev/raid/mps/mps.c:1450-1452, 1467-1469).

Impact

1. Stock DragonFly GENERIC (X86_64_GENERIC ships options INVARIANTS, sys/config/X86_64_GENERIC:56, and the guest kernel is built from it): KKASSERT is live (sys/sys/systm.h:98-101). The first SCSI IO timeout on any mps(4) adapter fires the callback from softclock without the lock → deterministic panic("assertion \"lockowned(&sc->mps_lock)\" failed in mpssas_scsiio_timeout"). A single slow/hung/recovering disk (cable glitch, error recovery, spin-up under load) converts an ordinary CAM timeout into a kernel panic.

2. Non-INVARIANTS builds: the KKASSERT is a no-op and the entire callback runs concurrently with the interrupt thread's lock-held completion paths: the lock-free mps_intr_locked() races the real interrupt handler over sc->replypostindex, reply descriptors and command completion (double completion of one reply → double xpt_done/double free), and the TAILQ_INSERT_TAIL(&targ->timedout_commands, ...) at mps_sas.c:1522 races mpssas_scsiio_complete's TAILQ_REMOVE at mps_sas.c:1826 — list corruption / use-after-free. (DragonFly's callout_stop() is synchronous (_callout_cancel_or_stop(..., sync=1), sys/kern/kern_timeout.c), so the completion side does wait for an already-running callback; the corruption vector is the callback body racing other mps_lock-protected state, which callout_stop cannot help.)

The sibling driver got this right: sys/dev/raid/mpr/mpr.c:1539 uses callout_init_lk(&cm->cm_callout, &sc->mpr_lock) and mprsas_scsiio_timeout/mprsas_tm_timeout assert (never take) the lock. The mps port kept the assert-style callback but not the lock-tied callout.

Trigger

Unprivileged local user performs IO to an mps(4) disk that stalls past the CCB timeout (failing device, error recovery, heavy load during link retraining). No privilege needed beyond ordinary disk access; the timeout itself is the trigger. Hardware-gated on an LSI SAS2008+ HBA being present.

Build/run

Nothing compilable without the HBA; run.sh records the guest hardware gate (driver compiled in — pci/mps — but no SAS2 PCI device, SIM never attaches). Source-level certainty is total (see VERDICT.md).

Fix

callout_init_lk(&cm->cm_callout, &sc->mps_lock) (mirror mpr.c:1539) and make mpssas_tm_timeout assert the lock instead of taking it (an AUTOLOCK callout already holds it; taking mps_lock again would self-deadlock). See fix.diff.

DF-3892 — VERDICT

Status: not_reproduced (hardware-gated) — defect certain from source. Impact class (on real hardware): kernel panic / DoS; race-based memory corruption on non-INVARIANTS builds.

Why it is real (path:line)

1. sys/dev/raid/mps/mps.c:860 — callout_init_mp(&cm->cm_callout); for every command in the pool. _callout_init_mp sets only CALLOUT_DID_INIT|CALLOUT_MPSAFE (sys/kern/kern_timeout.c:767-771).
2. sys/dev/raid/mps/mps_sas.c:1768-1769 — every XPT_SCSI_IO arms mpssas_scsiio_timeout on that callout.
3. sys/kern/kern_timeout.c:626-631 — a CALLOUT_MPSAFE callback is executed as the "Normal call": c->qfunc(c->qarg); with no lock acquired. Only CALLOUT_AUTOLOCK (from callout_init_lk, kern_timeout.c:609-625) wraps the callback in lockmgr(c->lk, ...).
4. sys/dev/raid/mps/mps_sas.c:1490 — the callback's second statement is KKASSERT(lockowned(&sc->mps_lock));. In softclock context this is always false.
5. sys/sys/systm.h:94-101 — KKASSERT is a real panic() whenever INVARIANTS is configured, and DragonFly's stock sys/config/X86_64_GENERIC includes options INVARIANTS (line 56); the audit guest kernel (and every default DF install) has it live. ⇒ first SCSI IO timeout on mps(4) = deterministic panic: mpssas_scsiio_timeout is entered, cm->cm_sc is read, the assert fires.
6. On builds without INVARIANTS the same callback body runs concurrently with lock-held paths: - mps_sas.c:1499 calls mps_intr_locked(sc) lock-free while the interrupt thread runs the identical function under mps_lock (mps.c:1450-1452): both advance sc->replypostindex, consume reply descriptors and run completions → double completion of one reply. - mps_sas.c:1520-1522 mutates cm->cm_ccb->ccb_h.status, cm->cm_state, and TAILQ_INSERT_TAIL(&targ->timedout_commands, ...) while mpssas_scsiio_complete (mps_sas.c:1800-1826, under the lock) does callout_stop + TAILQ_REMOVE(&targ->timedout_commands, ...)
   • mps_free_command → TAILQ corruption / use-after-free of the command and its CCB. Note DragonFly's callout_stop() is synchronous (sys/kern/kern_timeout.c: callout_stop → _callout_cancel_or_stop(cc, CALLOUT_STOP, 1); waiters loop until the request is honored), so the completion side does wait out an already-entered callback — the corruption vector is strictly the callback's own unsynchronized accesses, which is why the INVARIANTS panic (point 5) is the practically certain manifestation.
7. Reference implementation proving intent: the sibling mpr driver initializes callout_init_lk(&cm->cm_callout, &sc->mpr_lock) (sys/dev/raid/mpr/mpr.c:1539) and its mprsas_scsiio_timeout/mprsas_tm_timeout only assert lock ownership (sys/dev/raid/mpr/mpr_sas.c:1672). mps_sas.c was written in the same assert style (KKASSERT(lockowned(...)) also at mps_sas.c:901, 1568, 1801, 2762, 2797) but its command callouts were never lock-tied. mpssas_tm_timeout (mps_sas.c:1124-1133) even takes mps_lock() itself — correct for a lock-free callout and further proof the two timeout handlers were written against inconsistent assumptions.

Why it was not reproduced on this guest

run.sh / env.txt: the guest kernel contains the mps driver (kldstat -v shows 143 pci/mps) but QEMU provides no LSI SAS2 PCI device, so mps_attach never runs, no CAM SIM registers, and no mps command (hence no cm_callout timeout) can ever be armed. Reproduction requires: real SAS2 HBA + a device IO that exceeds its CCB timeout (e.g. dd against a da(4) LUN while the target is stalled), expected result on a GENERIC (INVARIANTS) kernel: panic: assertion "lockowned(&sc->mps_lock)" failed in mpssas_scsiio_timeout.

Exploitability

Panic is the guaranteed outcome on stock kernels (A:C high for an attacker with a way to stall a disk; DoS). On custom non-INVARIANTS kernels the callback/interrupt race is a genuine memory-corruption primitive (list corruption of timedout_commands, double completion), but it is narrow and hardware-gated; uid=0 escalation was not pursued because the guest cannot attach the hardware at all.

Fix validation

Not testable on this guest (no HBA). fix.diff mirrors the mpr(4) implementation exactly (callout_init_lk + assert-style mpssas_tm_timeout); it compiles by inspection against the same constructs used in mpr.c.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.