DF-3922 — hptmv 20 s command timeout vs completion: AB-BA wedge (callout_stop sleeping while holding driver_lock while hpt_timeout waits for that lock)

What

Every hptmv I/O arms a 20 s callout on the CCB's own callout handle: - sys/dev/raid/hptmv/entry.c:2669 (hpt_io_dmamap_callback) and entry.c:2865-2866 (OsSendCommand, CAM_SCATTER_VALID path): callout_reset(ccb->ccb_h.timeout_ch, 20*hz, hpt_timeout, ccb); - hpt_timeout (entry.c:2625-2632) first takes driver_lock: lock_driver(); fResetVBus(_VBUS_P0); unlock_driver(); - completion runs under driver_lock (all CheckPendingCall sites: hpt_intr:2123-2132, hpt_action:2273-2286, worker thread 2408-2428) and calls callout_stop(ccb->ccb_h.timeout_ch) from fOsCommandDone (entry.c:2910).

On DragonFly, callout_stop() is the synchronous variant: sys/kern/kern_timeout.c:1091-1094 → _callout_cancel_or_stop(cc, CALLOUT_STOP, 1), which at kern_timeout.c:910-921 sleeps in ssleep(c, &c->spin, 0, "costp", 0) with no timeout until the in-progress callback returns.

Deadlock

If the 20 s callout begins executing (callback INPROG, not yet through lock_driver()) at the instant a completion path holding driver_lock reaches callout_stop(): - CPU0: holds driver_lock, sleeps forever in costp waiting for the callout to finish; - CPU1: hpt_timeout blocks forever in lockmgr(&driver_lock, LK_EXCLUSIVE); - driver_lock is never released → every later hptmv entry (ioctl, CAM I/O, DPC worker, ISR via ithread) blocks on it → controller and its CAM devq wedge permanently. hpt_action is entered under sim_mplock (entry.c:2046-2047 passes &sim_mplock to cam_sim_alloc), so the giant CAM MP-lock is also held by any thread blocked entering this SIM - a system-wide storage wedge, not just one card.

Classic timeout-handler/lock-order inversion (CWE-667); the standard fix in SCSI LLDDs is to not call the synchronous stop under the lock, or to make the timeout handler take the lock via trylock/requeue.

Reproduce (needs RR1820 hardware + long-lived IO + timing luck)

Sustained I/O with one command completing within the microseconds around its 20-second timeout. Race window is per-command tiny; over months of uptime on a busy controller it is a plausible wedge. Cannot be triggered on the audit guest (no hardware).

Fix

--- a/sys/dev/raid/hptmv/entry.c
+++ b/sys/dev/raid/hptmv/entry.c
@@ -2626,7 +2626,15 @@ hpt_timeout(void *arg)
 static void
 hpt_timeout(void *arg)
 {
-   _VBUS_INST(&((PBUS_DMAMAP)((union ccb *)arg)->ccb_adapter)->pAdapter->VBus)
-   lock_driver();
-   fResetVBus(_VBUS_P0);
-   unlock_driver();
+   /* must not sleep waiting for driver_lock while a completion path may
+    * be sleeping in callout_stop() holding it (AB-BA wedge).  Requeue
+    * until the lock is free; fOsCommandDone's callout_stop cancels us. */
+   for (;;) {
+       _VBUS_INST(&((PBUS_DMAMAP)((union ccb *)arg)->ccb_adapter)->pAdapter->VBus)
+       if (lockmgr(&driver_lock, LK_EXCLUSIVE | LK_NOWAIT) == 0)
+           break;
+       callout_reset(((union ccb *)arg)->ccb_h.timeout_ch, hz, hpt_timeout, arg);
+       return;
+   }
+   fResetVBus(_VBUS_P0);
+   unlock_driver();
 }

(see fix.diff for the exact tree-accurate hunk)

DF-3922 VERDICT

status: not_reproduced (hardware-gated race) | impact observed: none (potential: permanent kernel/storage wedge) | source certainty: likely

Mechanics verified at source level

1. callout_stop() on DragonFly is synchronous: kern_timeout.c:1091-1094 passes sync=1; _callout_cancel_or_stop (kern_timeout.c:859-930) at :910-921 sleeps in ssleep(..., "costp", 0) — no timeout — while the callback is CALLOUT_INPROG.
2. hpt_timeout (entry.c:2625-2632) executes lock_driver() as its first action and only then resets the bus.
3. fOsCommandDone (entry.c:2910) calls callout_stop(ccb->ccb_h.timeout_ch) while running under driver_lock (every CheckPendingCall/DPC execution context in this driver holds it: hpt_intr 2123-2132, hpt_action 2273-2286, worker 2408-2428).
4. => if the callout fires concurrently with a completion holding the lock, CPU0 sleeps in "costp" holding driver_lock, CPU1 sleeps in lockmgr waiting for driver_lock. Neither has a timeout. controller + DPC worker + all CAM entries wedge; because the SIM uses &sim_mplock (entry.c:2047), the giant CAM MP lock serializes other legacy SIMs against blocked entrants.

Why "likely" and not "certain"

The window requires the 20 s callout to begin executing in the exact microseconds a completion under driver_lock reaches callout_stop. The mechanism is certain (verified code), the manifestation is a rare race — hence confidence "likely". Hardware-gated: cannot be raced on the audit guest (no RR1820).

Note

hptmv also reuses the CCB's CAM-owned callout (ccb_h.timeout_ch) for its private timeout — CAM allocates/frees this callout (cam_xpt.c:4934-4945); the stop/start dance here is the only synchronization, which is what makes the synchronous-stop semantics load-bearing for correctness.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.