DF-4613 — ips_timeout forces adapter reinit while holding queue_lock and then sleeps forever on a completion only the (lock-blocked) interrupt handler can deliver

What

sys/dev/raid/ips/ips.c:331-371 (ips_timeout), ips.c:355 (forced reinit), ips.c:585 / ips.c:769 (interrupt handlers take the same lock), sys/dev/raid/ips/ips_commands.c:722,743 (ips_timed_wait(..., 0) = infinite tsleep) via ips_clear_adapter() called from ips_morpheus_reinit(sc, 1) (ips.c:516) / ips_copperhead_reinit(sc, 1) (ips.c:738).

Timer-path twin of DF-2163 (which filed the identical pattern reached from the IPS_USER_CMD ioctl in ips_ioctl.c:125-139). This path needs no user action: it fires automatically whenever any card command is stuck for ~100 s.

Sequence (morpheus; copperhead is identical in shape)

1. A command hangs on the card. ips_timeout (10 s callout, softclock thread) decrements timeout each tick; at 0 it sets IPS_TIMEOUT, error-completes the command via its callback (ips.c:343-349) — all under lockmgr(&sc->queue_lock, LK_EXCLUSIVE) taken at ips.c:337.
2. Once no command still has a pending timeout (state == 0), ips.c:354-355 calls sc->ips_adapter_reinit(sc, 1) still holding queue_lock.
3. ips_morpheus_reinit(force=1) resets the card (5 s DELAY + up to 45+240 1-second DELAY busy-waits — themselves under the lock, starving every other callout on the CPU), unmasks interrupts (ips_write_4(OIMR, 0), ips.c:515), then ips_clear_adapter(sc) (ips.c:516).
4. ips_clear_adapter → ips_send_config_sync_cmd (ips_commands.c:707-725) issues the sync command and calls ips_timed_wait(command, "ipssyn", 0) — timo = 0 = sleep forever (ips_commands.c:43-58; plain tsleep, which does not release lockmgr locks).
5. The card completes the command → IRQ → ips_morpheus_intr → lockmgr(&sc->queue_lock, LK_EXCLUSIVE|LK_RETRY) (ips.c:585) — blocks forever: the sleeping softclock thread owns the lock.
6. The wakeup never happens; the softclock thread never releases the lock; every subsequent IPS I/O, every ioctl, and every future ips_timeout tick blocks on queue_lock. Permanent wedge of the controller plus two kernel threads.

LK_CANRECURSE (lockinit at ips_pci.c:154) does not help: the interrupt handler is a different thread.

Build / run

Requires an IBM ServeRAID (ips) controller: PCI ids 0x1014:0x01BD (morpheus), 0x1014:0x002E / 0x1014:0x0250 / Adaptec 0x9005 (copperhead/marco). The audit guest is QEMU/virtio and has no such device (env.txt); QEMU has no ServeRAID device model, so the path cannot be instantiated. Verdict rests on the static trace above (certain) plus the mechanically identical DF-2163 pattern.

Expected behavior (if hardware were present)

~100 s after any command hangs, the console prints "WARNING: command timeout… toaster mode", then "resetting adapter…"; when the reset finishes and ips_clear_adapter issues the config sync, the system wedges: ps shows the softclock thread in tsleep ("ipssyn") holding ipslk and the IPS ithread blocked on ipslk; all I/O to ips(4) logical drives hangs forever; boot device on ips ⇒ hard hang requiring power cycle.

Contents

• VERDICT.md, env.txt, fix.diff, manifest.json, verdict.json

VERDICT — DF-4613

Status: not_reproduced (no IBM ServeRAID controller on the audit guest — env.txt; QEMU provides no device model for 0x1014:0x01BD/0x002E/0x0250 or 0x9005 ServeRAID). The deadlock is proven by line-accurate static trace with certain confidence; it is the mechanically identical twin of the verified pattern in DF-2163 (ioctl path), reached here from the watchdog timer with no user interaction.

Proof chain (every hop cited)

1. The lock is held across the whole recovery. ips_timeout (ips.c:337): lockmgr(&sc->queue_lock, LK_EXCLUSIVE|LK_RETRY) … ips.c:355: if (sc->ips_adapter_reinit(sc, 1)) — still inside the critical region (released only at ips.c:370).

2. The reinit sleeps waiting for card completion. ips_morpheus_reinit(sc, force=1) → ips_clear_adapter(sc) (ips.c:516; copperhead: ips.c:738) → ips_send_config_sync_cmd → ips_timed_wait(command, "ipssyn", 0) (ips_commands.c:722); second wait in ips_send_error_table_cmd (ips_commands.c:743). ips_timed_wait loops tsleep(&command->completed, 0, id, timo) (ips_commands.c:47-56) — timo 0 = no timeout. DragonFly tsleep does not release lockmgr locks (only lksleep does).

3. The only wakeup source needs the same lock. command->completed = 1; wakeup(&command->completed) happens solely in ips_wakeup_callback (ips_commands.c:66-72), which runs from ips_morpheus_intr / ips_copperhead_intr — both of which first take lockmgr(&sc->queue_lock, LK_EXCLUSIVE|LK_RETRY) (ips.c:585, ips.c:769) before draining completions and invoking callbacks (ips.c:573, ips.c:775).

4. Cross-thread ⇒ CANRECURSE is irrelevant. queue_lock is lockinit(..., LK_CANRECURSE) (ips_pci.c:154); recursion only helps the owning thread. The interrupt handler is a separate ithread → it blocks, the sleeper never wakes. Softclock handlers are lwkt threads (sys/kern/kern_timeout.c:416-423), so the tsleep is mechanically legal and simply never terminates.

5. Additional degradation while the lock is held: morpheus reinit busy waits DELAY(5000000) + up to 45×1 s + 240×1 s (ips.c:487-507) with the lock held in the softclock thread — starving every other callout on that CPU for minutes before the final deadlock even forms.

DF-2163 documented the same lock/sleep inversion entered from ips_ioctl_cmd (ips_ioctl.c:125-139, tsleep loop) and noted ips_timeout would be blocked by that hang. DF-4613 is the distinct ips.c entry: the watchdog itself creates the inversion, no ioctl and no privileges required — the trigger is purely "a card command did not complete for ~100 s" (failing hardware, torn cable, firmware stall), exactly the situation the watchdog exists to recover from. The recovery path cannot succeed by construction: the config-sync it issues can never complete while its own lock is held.

Impact

Permanent denial of service of the storage controller and two kernel threads (softclock thread of that CPU + IPS ithread). All I/O to ips(4) volumes hangs; if the root filesystem is on an ips drive, the whole system hangs. Requires a hung card command to trigger (hardware fault class), hence Medium (local DoS, unusual precondition) rather than High.

Fix

Drop queue_lock before calling ips_adapter_reinit in ips_timeout (lockmgr(&sc->queue_lock, LK_RELEASE) at ips.c:355, re-acquire after), and give the ips_timed_wait calls in ips_clear_adapter's helpers a finite timeout so a dead card cannot sleep forever. See fix.diff.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.