β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-3607

ncr(4): timeout daemon completes an nccb still armed in the chip's DSA β€” later phase-mismatch resumes DMA from stale/reused scatter descriptors (memory corruption) or wedges the HBA

Summary

ncr_timeout() cannot distinguish 'queued, never selected' from 'connected, mid-phase' commands - both carry host_status HS_BUSY. It patches only the start-queue case and then unconditionally completes the nccb. DragonFly CAM never arms ccb_h.timeout_ch and ncr's XPT_ABORT/XPT_RESET_DEV return CAM_REQ_INVALID, so this daemon is the only timeout enforcement. After completion cp->ccb==NULL/magic cleared, but the chip's DSA still names the nccb and it remains reusable. A later phase mismatch reaches ncr_int_ma(), which finds the dead/reused nccb by physical DSA match (the mismatch sanity check's return is commented out) then either wedges the HBA or builds cp->patch[] from the stale/reused scatter table and resumes host DMA into a buffer the CAM periph already released (use-after-free DMA write). Any SCSI target that stalls in a data phase past the CCB timeout then resumes - malfunctioning or malicious device. RUNTIME: ncr0 brought fully LIVE on QEMU's emulated LSI 53C895A (kernel rebuilt with sym's probe neutered after discovering newbus's larger-probe-value-wins rule makes stock GENERIC always give the chip to sym); attach+snooptest passed, the daemon completion path observed firing 90+ times - but the emulated target cannot be made to stall mid-phase on demand, so the connected-job window was not enterable (inconclusive; source trace line-accurate). Fix: DSA-check in ncr_timeout + magic/ccb-check in ncr_int_ma.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-3607 Β· 8 files
FileTypeDescriptionSize
README.md β€” 1.1 KB ↓ raw
VERDICT.md β€” 6.4 KB ↓ raw
ncr_timeout_race.c β€” 3.0 KB view raw
fix.diff β€” 1.8 KB view raw
build.log β€” 307 B view raw
run.log β€” 1.9 KB view raw
env.txt β€” 337 B view raw
verdict.json β€” 3.7 KB view raw

DF-3607 β€” ncr(4): CCB timeout daemon completes an nccb still armed in the chip's DSA; a subsequent phase-mismatch resumes DMA from stale/reused scatter descriptors (kernel memory corruption) or wedges the HBA (DoS).

Reproduce (requires NCR 53C8xx PCI HBA β€” 53c810..53c895A β€” with an attached SCSI target that can stall past the CCB timeout):

build: cc -O2 -o ncr_timeout_race ncr_timeout_race.c (in guest, root) run: ./ncr_timeout_race pass0 200 (in guest, root) expect: with a stalling target β€” console "timeout nccb=0x... (skip)" from ncr.c:5162 followed (on target resume) by a phase-mismatch handled against the completed nccb in ncr_int_ma (ncr.c:5640): either a silent chip halt (all further I/O hangs) or a resumed transfer DMAing into the released/reused data buffer.

On the audit QEMU guest the emulated lsi53c895a target is well-behaved and cannot be made to stall mid-phase, so the full race cannot be choreographed there; see VERDICT.md for the source-level trace and what WAS executed.

VERDICT.md
↓ download raw

DF-3607 β€” VERDICT

Finding: ncr(4)'s software timeout daemon (ncr_timeout) completes an nccb that is still armed in the 53C8xx script processor's DSA. A subsequent phase-mismatch on that connection is fixed up in ncr_int_ma() against the dead (or already re-used) nccb: the chip is either wedged (returns without restart β€” all further I/O on the HBA hangs) or restarted with a patch built from the nccb's stale/re-used scatter descriptors, resuming host DMA into a buffer the CAM periph has already released. The first form is a local DoS; the second is kernel memory corruption via DMA.

Status: inconclusive β€” reproduced at source level (line-accurate trace below); full runtime choreography requires an NCR 53C8xx HBA plus a SCSI target that stalls mid-phase past the CCB timeout and then resumes. The audit guest's QEMU-emulated 53C895A target is well-behaved and cannot be made to stall on demand, so the race window itself could not be driven to conclusion on the guest (attempts documented below).

Why the code is wrong β€” line-accurate trace (sys/dev/disk/ncr/ncr.c)

  1. ncr_action() XPT_SCSI_IO sets cp->host_status = HS_BUSY (ncr.c:4116) β€” and the host status stays HS_BUSY for the whole connected life of the command: the script only updates HS_REG (scr1) at completion-ish points (complete/status/cleanup). cp->tlimit = time_uptime + timeout/1000 + 2 (ncr.c:4137-4138).

  2. DragonFly CAM never arms ccb_h.timeout_ch β€” verified by grep: there is no callout_reset on timeout_ch anywhere under sys/bus/cam/ (only callout_init at cam_xpt.c:3906 and alloc/free). CAM-level recovery is also unavailable for this SIM: XPT_ABORT and XPT_RESET_DEV are unimplemented and return CAM_REQ_INVALID (ncr.c:4170-4179). The driver's own daemon is therefore the only timeout enforcement.

  3. ncr_timeout() (ncr.c:5101) walks all nccbs; for any with host_status != 0 and tlimit <= now it: - disables the reselect pattern (cp->jump_nccb.l_cmd = SCR_JUMP, ncr.c:5159) β€” protects the disconnected case; - patches the start-queue entry to skip only if the job was never selected (launch.l_paddr == select, ncr.c:5160-5166) β€” after a successful selection the script has already overwritten launch with the idle-jump (loadpos, script at ncr.c:1599-1601), so a connected, mid-transfer job gets no protection at all; - HS_BUSY/HS_NEGOTIATE/HS_DISCONNECT -> HS_TIMEOUT, cp->tag = 0, ncr_complete(np, cp) (ncr.c:5168-5181).

  4. ncr_complete() (ncb guarded at ncr.c:4428) sets cp->ccb = NULL (ncr.c:4452) and ncr_free_nccb clears the magic (ncr.c:6474) β€” but the nccb stays in np->link_nccb and the chip's DSA register still holds CCB_PHYS(cp, phys). The CAM periph gets the CCB back and may free/reuse the data buffer.

  5. The stalled target resumes/changes phase -> MA interrupt -> ncr_int_ma() (ncr.c:5640): - the nccb is found by physical DSA match regardless of magic (ncr.c:5692-5694); the sanity check that would notice the stale/dead nccb prints a mismatch but its return is commented out (ncr.c:5701-5707 /* return; */); - PRINT_ADDR(cp->ccb) with cp->ccb == NULL is harmless (xpt_print_path(NULL) is NULL-safe, cam_xpt.c) β€” no panic there; - on opcode mismatch it returns without restarting the chip (ncr.c:5775-5782): the script processor stays halted on the dead connection; every later I/O to this HBA hangs => local DoS; - on a data-phase match it builds cp->patch[] from the stale cp->phys scatter table β€” or, if ncr_get_nccb() has already handed the same nccb to a new XPT_SCSI_IO (freed nccbs are reused, ncr.c:6411-6454), from the new command's table at the old command's residual position β€” writes it (ncr.c:5798-5808) and restarts the script processor at dispatch/checkattn (ncr.c:5824-5828). The chip then DMAs residual data into whatever the stale (or mismatched new) descriptors point at: a buffer the periph released or reused => use-after-free DMA write.

The same completion-while-armed pattern exists for HS_DISCONNECT (disconnected, never-resumed jobs are safe to complete because the reselect chain no longer matches them) β€” the unprotected state is exactly the connected one.

What was executed on the guest

  • Guest kernel X86_64_GENERIC contains the driver (nm /boot/kernel/kernel: ncr_action, ncr_timeout, ncr_int_ma, ncr_attach present; 45 ncr symbols).
  • QEMU lsi53c895a added on the guest PCI bus (device 1000:0012 = NCR_895A_ID in ncr.c:1331). Default boot: sym(4) wins the probe (sym returns BUS_PROBE_DEFAULT=-20; newbus keeps the numerically greater probe result, sys/kern/subr_bus.c:1456 result > pri), so ncr never attaches out of the box.
  • One-line guest kernel rebuild (sym_conf.h SYM_SETUP_LP_PROBE_MAP 0x04 β€” the documented mechanism, sym_conf.h:270-292) makes sym probe the 895A at low priority so ncr claims it. See build.log / env.txt.
  • With the rebuilt kernel the ncr driver attaches to the emulated HBA and the CAM scan drives real XPT_SCSI_IO traffic through ncr_action (see run.log). The emulated scsi-hd target completes every phase immediately; no mid-phase stall is available from -device scsi-hd, so the timeout daemon never fires on a connected job in these runs β€” the race window could not be entered with stock QEMU devices. The trigger program (ncr_timeout_race.c) documents how a programmable / stalling target would drive it (1 ms CCB timeouts against a slow target).

Bottom line

Source-certain lifetime bug in the driver's only timeout/recovery path; concrete kernel-memory-corruption primitive (stale scatter descriptors programmed into the chip after the owning CCB was completed) gated on a stalling SCSI target behind an NCR 53C8xx HBA. Recommend the fix.diff (reject completions while DSA still names the nccb β€” force the hard-recovery path; never fix up a phase mismatch against a dead nccb).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

not_testable
baseline no→ patch + rebuild →patched clean

fix.diff authored (two guards: connected-ccb timeout -> bus reset; dead-nccb phase fixup -> bus reset). Baseline condition (stalling target mid-phase) is not reproducible on the QEMU lsi emulator, so the patched-kernel A/B could not be run; not_testable.

findings/poc/DF-3607/fix.diff
↓ fix.diffper-fix-DF-3607

Confirmed kernel references

Detail

Evidence (decisive lines)

VERDICT.md (line-accurate trace); run.log (daemon path live on guest, attach + COMMAND FAILED + timeout nccb lines); env.txt (guest uname + binaries); fix.diff; ncr_timeout_race.c

PoC changes

replaced cam_errstr with cam_errbuf (DFly camlib); dropped scsi_read_write()/CCB_CLEAR_ALL_EXCEPT_HDR (not in DFly userland headers) for a manually built 10-byte READ CDB; kept timeout=1ms per the sketch

Verified recommended fix

In ncr_timeout, do not soft-complete an nccb whose CCB_PHYS still equals the chip DSA - force ncr_init recovery; in ncr_int_ma, refuse to fix up (hard reset) when the matched nccb has magic != CCB_MAGIC or ccb == NULL

Verdict

ncr_timeout() (ncr.c:5148-5182) completes nccbs that are still armed in the chip's DSA when a connected command exceeds its CCB timeout; DragonFly CAM never arms ccb_h.timeout_ch and ncr's XPT_ABORT/XPT_RESET_DEV are unimplemented, so this daemon is the only recovery path. A later phase mismatch reaches ncr_int_ma() with the dead (or reused) nccb found by DSA match (ncr.c:5692-5707, mismatch-return commented out): the chip is either wedged (ncr.c:5775-5782, all HBA I/O hangs) or restarted from stale/mismatched scatter descriptors (ncr.c:5798-5828) -> DMA into released buffers (kernel memory corruption). Runtime: driver made fully live on the audit guest (ncr0 attached to QEMU-emulated LSI 53C895A after a one-line guest-side sym probe change; snooptest passed; CAM probes flowed through ncr_action; the daemon's completion path observed firing 90+ times). The race window itself needs a SCSI target that stalls mid-phase past tlimit and then resumes; stock QEMU scsi devices cannot stall on demand, so the trigger binary (builds clean, /tmp/ncr_timeout_race) could not be aimed at a pass device (no INQUIRY completes through the chip on the emulator). Source-certain; runtime choreography not enterable on this guest.