DF-3982 — sbp(4): synchronous callout_stop() from hard-interrupt context sleeps in costp (DF-3715 family, sbp.c instance)

Where

sys/dev/disk/sbp/sbp.c — three callout_stop() sites on the receive path:

• sbp_recv1() sbp.c:1600 — callout_stop(&target->mgm_ocb_timeout) when a matching management-ORB status arrives,
• sbp_dequeue_ocb() sbp.c:2590 — callout_stop(ocb->ccb->ccb_h.timeout_ch) for every matched command ORB (called from sbp_recv1:1605),
• sbp_abort_ocb() sbp.c:2705 — same, via sbp_abort_all_ocbs from sbp_recv1 dead-agent handling (sbp.c:1663-1669 → sbp_agent_reset sbp.c:1080) and from sbp_probe_target (sbp.c:688).

All three execute in the fwohci hard-interrupt path: FWOHCI_TASKQUEUE=0 (sys/bus/firewire/fwohcivar.h:37) so fwohci_intr (fwohci.c:2009) runs fwohci_intr_body directly (fwohci.c:2038) → fwohci_arcv under crit_enter() (fwohci.c:2688) → fw_rcv (fwohci.c:2833) → fw_xfer_done → xfer->act.hand (firewire.c:991) = sbp_recv, which adds its own crit_enter() (sbp.c:1823).

DragonFly's current callout implementation made callout_stop() synchronous: _callout_cancel_or_stop(cc, CALLOUT_STOP, sync=1) (kern_timeout.c:1091, 858-927) ssleep()s on costp (kern_timeout.c:923) whenever the callout is queued or INPROG on any CPU, until the callback completes. Sleeping from a device interrupt handler (and inside a critical section) is illegal: the fwohci IRQ stalls for the whole duration of sbp_mgm_timeout's body (kprintf + sbp_reset_start → sbp_write_cmd → fw_asyreq, which allocates and can be delayed by bus retries — DF-1021 family), and the INPROG-blocked softclock thread additionally starves every tsleep-timeout on its CPU (observed cascade).

Reachability

A hostile FireWire device knows its ORB addresses (the initiator wrote them to its management agent) and can time/spam status writes; landing one while the 5 s mgm_ocb_timeout (sbp.c:1351) fires — or while a CCB timeout callback runs — puts the interrupt thread to sleep. Race-window gated but sprayable by the attacker.

Evidence in this pack (mechanism PROVEN on the shipped guest kernel)

run.log (live capture):

1237 D5  costp  sysctl        <- callout_stop() caller asleep INSIDE crit (the sbp.c:1600 state)
 -1 12  B2  bcb  softclock 2   <- mgm-timeout-shaped callback INPROG
1256 D2  wkdelay sysctl        <- cascade: unrelated tsleep-timeout starved by the wedged softclock

(sbp_replica.c mechanism B; negative control in RUN 1: stop of a non-INPROG callout returns in 9 ms.)

Reproduce

cd /root/kmod && make && kldload ./sbpprep.ko
sysctl kern.sbpprep_armb=1
(sysctl -n kern.sbpprep_stopb=1 >/dev/null 2>&1 </dev/null &)
sleep 3
ps -o pid,stat,wchan,comm -ax | grep sysctl   # expect D-state, wchan costp
ps -axH -o pid,tdpri,stat,wchan,comm | grep softclock   # expect B2 bcb

Fix

Interrupt path must not use the synchronous stop. fix.diff switches the three sites to callout_stop_async() (the semantics this pre-rewrite driver was written against) and closes the resulting completion race in sbp_mgm_timeout by re-checking target->mgm_ocb_cur == ocb under crit before touching the ocb. Full correctness eventually wants a lock redesign of the mgm queue, as noted in the finding.

DF-3982 VERDICT — blocking callout_stop() in sbp's hard-interrupt path

Status: inconclusive end-to-end (sbp trigger hardware-gated), mechanism PROVEN live on the shipped guest kernel (blocked caller in D5 costp inside a critical section + INPROG softclock thread in B2 bcb + cascade to an unrelated wkdelay sleep). Source-level certainty: certain.

Root cause

1. sbp_recv1 runs in the fwohci hard-interrupt path under crit_enter() — chain: fwohci_intr (fwohci.c:2009) → fwohci_intr_body (fwohci.c:2038, FWOHCI_TASKQUEUE=0 per fwohcivar.h:37) → fwohci_arcv crit_enter() (fwohci.c:2688) → fw_rcv (fwohci.c:2833) → fw_xfer_done → act.hand (firewire.c:991) → sbp_recv crit_enter() (sbp.c:1823).

2. On that path the driver calls the synchronous callout_stop() at - sbp.c:1600 (mgm_ocb_timeout, on mgm-ORB status match), - sbp.c:2590 (CCB timeout, in sbp_dequeue_ocb — called at sbp.c:1605), - sbp.c:2705 (CCB timeout, in sbp_abort_ocb via sbp_abort_all_ocbs from sbp_recv1's dead-agent branch sbp.c:1663-1669 → sbp_agent_reset sbp.c:1080, and from sbp_probe_target sbp.c:688).

3. DragonFly's current callout_stop() (kern_timeout.c:1091 → _callout_cancel_or_stop(cc, CALLOUT_STOP, sync=1), kern_timeout.c:858) sleeps in ssleep(..., "costp", 0) (kern_timeout.c:923) while the callback is queued or INPROG on any CPU — i.e. the fwohci interrupt thread is put to sleep, with a critical section held, for the full duration of sbp_mgm_timeout's body (sbp.c:2126-2146: kprintf, sbp_free_ocb, sbp_reset_start → sbp_write_cmd → fw_asyreq, which allocates — see DF-1021 for the M_WAITOK-in-crit tlabel path) or of sbp_timeout's body (xpt_freeze_devq, abort-all). The old-BSD-callout semantics this 2003-era driver was written for (stop = async cancel) no longer hold.

4. Cascade (observed): the softclock thread blocked INPROG inside the callback cannot service that CPU's wheel, so unrelated tsleep-timeouts starve — a third thread stuck in D2 wkdelay was captured live while trying to release the experiment.

What was run on the guest (DF 6.5-DEVELOPMENT, X86_64_GENERIC)

Mechanism replica B (sbp_replica.c, this pack): a 1-tick mgm_ocb_timeout-shaped callout whose body blocks until released (the unbounded-body shape of sbp_mgm_timeout), plus a sysctl handler that performs crit_enter(); callout_stop(&c); — the exact state of sbp.c:1600 — from a detached process:

 1237 D5     costp    sysctl      <- callout_stop caller: D-state, wchan 'costp', crit held
   -1 12 B2  bcb      softclock 2 <- target callout: INPROG, blocked in body
 1256 D2     wkdelay  sysctl      <- cascade: tsleep timeout starved (wedged wheel)

Negative control (RUN 1): callout_stop() on a non-INPROG callout returned in 9 ms — the block is specifically the INPROG/queued wait of kern_timeout.c:888-923. Both behaviors match the source.

Why not reproduced for the finding itself: the sbp trigger needs FireWire hardware plus an attacker-timed status/timeout collision; the guest has no OHCI-1394 controller (env.txt). The blocking mechanism itself was demonstrated on the very kernel the audit targets.

Not-a-duplicate check

DF-3715/DF-3738 (em/emx) and DF-3760 (alc) prove the same kernel-wide callout_stop semantics in serializer deadlocks of net drivers; this is the sbp.c instance: different driver, different context (hard-IRQ receive path, crit-only, no serializer), different victim (fwohci IRQ thread + softclock wheel starvation). No prior sbp.c finding covers the callout sites.

Fix

fix.diff: callout_stop_async() at sbp.c:1600/2590/2705 (interrupt-path semantics of the era the driver was written for) + sbp_mgm_timeout re-validating target->mgm_ocb_cur == ocb under crit before acting, so the now-unstoppable callback cannot race the recv1 completion into a double sbp_free_ocb of the same ocb. A complete fix is a lock/timeout redesign of the mgm queue (flagged in the finding as follow-up).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.