ahci_xpt_action() XPT_SCSI_IO re-entrant probe runs without the port lock β concurrent probe/state-machine race corrupts port state
Summary
The XPT_SCSI_IO handler calls ahci_cam_probe while holding only ap_sim_lock. Every other caller of ahci_cam_probe and every other hardware-touching branch of ahci_xpt_action holds ap->ap_lock. The probe issues polled commands mutating ap_active/ap_sactive/ap_run_flags and CI/SACT/FBS registers - the file's own NOTE-1 documents that concurrent ahci_port_intr execution creates 'a huge mess in the sequencing of the chipset'. Trigger: hotplug/error event on a PM target sets NEED_IDENT and the state machine re-probes under ap_lock while a queued XPT_SCSI_IO re-probes unlocked - unsynchronized ccb-pool KKASSERTs, double-processed completions tripping the serial guard (panic), corrupted lists -> lost completions/timeout storms, worst case one ccb slot with two simultaneous owners (DMA command-table cross-write). Requires SATA PM hardware (absent in QEMU; attach attempts hang in firmware - documented). Lock-bracket fix mirroring the dispatch 60 lines below.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4203 Β· 9 files| File | Type | Description | Size | |
|---|---|---|---|---|
| race.sh | β | 1.4 KB | view raw | |
| run.log | β | 1.9 KB | view raw | |
| panic-race.txt | β | 2.5 KB | view raw | |
| panic-race2.txt | β | 980 B | view raw | |
| env.txt | β | 2.1 KB | view raw | |
| fix.diff | β | 2.0 KB | view raw | |
| VERDICT.md | β | 4.5 KB | β raw | |
| manifest.json | β | 1.3 KB | view raw | |
| verdict.json | β | 4.1 KB | view raw |
DF-4203 VERDICT β unlocked re-entrant probe in XPT_SCSI_IO (ahci_cam.c)
Bottom line (2026-10-15 re-verification with live AHCI hardware)
Race consequence REPRODUCED twice on live hardware (trigger state
injected); natural arming remains hardware-gated (SATA PM). With a real
AHCI controller + disk on the guest, forcing at->at_probe =
ATA_PROBE_NEED_IDENT (the state whose ONLY production producer is the PM
hard-reset completion at ahci_pm.c:876) while normal I/O, device resets and
pass CCBs run concurrently, the unlocked re-probe at ahci_cam.c:989-990
panics the kernel within seconds in exactly the double-completion mode the
finding predicts. The identical load WITHOUT arming (control) survives
clean. The locking asymmetry itself (ahci_cam_probe called under
ap_sim_lock only) is unchanged in source.
1. What is new vs the first verification round
The first round (this directory, earlier VERDICT) correctly identified the
two-lock design, the single unlocked caller, and the PM-only arming problem,
and could not test: the guest had no AHCI controller and QEMU could not
boot with one. This round got live AHCI hardware by attaching -device
ahci + ide-hd/ide-cd at QEMU launch (up_ahci.sh; boot order pinned via
bootindex β the old firmware hang was an artifact of the earlier
invocation). ahci0.0 DISK (tags=32/32) = da0/pass0, ahci0.1 ATAPI =
cd0/pass1.
2. Live runs (race.sh, 2026-10-15)
Armed run = 4 concurrent dd readers on da0 + camcontrol reset da0 loop
(locked state-machine probes) + continuous pass XPT_SCSI_IOs + KLD arming
at_probe = NEED_IDENT continuously for 25-45s (df2420_rig.c arm
command β reaches the real port via devclass_find("ahci")).
Run 1 (45s arm) β panic in <45s (panic-race.txt):
- dmesg shows repeated concurrent probes (
ahci0.0: Found DISK ...many times) from the NEED_IDENT branch while dd IO completes. panic: assertion "xa->state != ATA_S_ONCHIP && xa->state != ATA_S_PUT" failed in ahci_ata_cmd_done at ahci.c:3776(the panic line is character-interleaved on the serial console with a concurrent probe'sFound DISKkprintf from another CPU β itself proof of unsynchronized concurrent execution).- Backtrace:
ahci_ata_cmd_done β ahci_ata_cmd_done β ahci_port_intr β ahci_port_thread_core β ahci_port_threadβ the port thread completed a ccb whose xa had already been PUT (double completion).
Run 2 (25s arm) β panic in <25s (panic-race2.txt), attribution is direct:
panic: assertion "ccb->ccb_xa.state != ATA_S_PUT" failed in ahci_put_ccb at ahci.c:3321
ahci_put_ccb() at ahci_put_ccb+0xde
ahci_put_ccb() at ahci_put_ccb+0xde <-- double put
ahci_ata_complete_disk_rw() at +0xec
ahci_port_intr() at ahci_port_intr+0x790
ahci_poll() at ahci_poll+0x50
ahci_cam_probe() at ahci_cam_probe+0xfa <-- THE UNLOCKED RE-PROBE
Debugger("panic")
The unlocked XPT_SCSI_IO re-probe polls a command whose completion was already processed elsewhere and puts the ccb back a second time β the finding's worst-case mode ("one ccb slot with two simultaneous owners"), caught by the serial/state guards.
Control (identical load, NO arming, 70s): no panic, no extra probes
(1 Found DISK line total = boot probe), 14,650 pass CCBs all clean,
guest healthy. The NEED_IDENT branch is the differentiator.
3. Honest limits
- The trigger state was injected by the rig. On unmodified hardware the branch is only armed by ahci_pm.c:876 (PM hard reset), and QEMU has no SATA PM device model. Natural reproduction therefore requires physical PM hardware; everything downstream of the arming is production code on production state.
- With the PM absent,
at->at_probefor a direct-attached disk is only GOOD/FAILED after boot, so the unlock can also not fire via disk hot-plug events on this guest.
4. Fix
fix.diff (now the combined validated diff; the DF-4203 hunk is the ap_lock bracket, identical to the first-round proposal): bracket the re-probe with ahci_os_lock_port()/ ahci_os_unlock_port(), mirroring the dispatch 60 lines below. On the patched kernel the identical armed race ran clean (no panic, probes serialized), while the control load stays healthy.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedfixed
fix-validation.log: patched kernel, identical armed race (25s arm) -> no panic, no assertions, 1518 Found-DISK re-probes (branch exercised, now serialized), guest healthy
Confirmed kernel references
Detail
Evidence (decisive lines)
panic-race2.txt (double-put backtrace through ahci_cam_probe), panic-race.txt (port-thread double completion + interleaved concurrent probe kprintf), run.log (race + clean control transcript), race.sh, fix-validation.log (patched kernel: armed race clean, 1518 probes, 0 panics)
PoC changes
first round (this pack) concluded untested: no AHCI controller and a firmware-level boot hang. This round: (1) got live AHCI hardware by attaching -device ahci + ide-hd/ide-cd at QEMU launch with explicit bootindex pinning (the old hang was an invocation artifact), (2) reused the DF-2420 rig's 'arm' command to force NEED_IDENT on the real port (no driver logic copied), (3) built race.sh combining dd readers + camcontrol reset loop + pass CCB storm + arming, plus a no-arming control
Verified recommended fix
bracket the re-entrant ahci_cam_probe() in the XPT_SCSI_IO path with ahci_os_lock_port()/ahci_os_unlock_port() (fix.diff, validated: armed race runs clean with the probe serialized)
Verdict
Race consequence REPRODUCED twice on live AHCI hardware (trigger state injected; natural arming remains PM-hardware-gated). With at->at_probe forced to ATA_PROBE_NEED_IDENT (its only production producer is the PM hard-reset completion ahci_pm.c:876; QEMU has no SATA PM device model) while dd readers, a camcontrol reset loop and a pass XPT_SCSI_IO storm run, the unlocked re-probe at ahci_cam.c:989-990 panics the kernel within 25-45 seconds in exactly the predicted double-completion mode: run 2 panic 'ccb->ccb_xa.state != ATA_S_PUT failed in ahci_put_ccb at ahci.c:3321' with backtrace ahci_cam_probe -> ahci_poll -> ahci_port_intr -> ahci_ata_complete_disk_rw -> ahci_put_ccb x2 (the unlocked probe polls a command already completed elsewhere and free-lists the ccb twice); run 1 panic 'xa->state != ATA_S_ONCHIP && xa->state != ATA_S_PUT in ahci_ata_cmd_done at ahci.c:3776' from the port thread, with the panic kprintf character-interleaved on the serial console with a concurrent probe's Found-DISK kprintf from another CPU. CONTROL: the identical load WITHOUT arming runs 70s clean (14650 pass CCBs, single boot probe, no panic) - the NEED_IDENT branch is the differentiator. Fix (ap_lock bracket around the re-probe, mirroring the dispatch 60 lines below) validated: the identical armed race on the patched kernel completed with 1518 serialized re-probes, zero panics/assertions. Caveat kept honest: had the trigger not been injected, the branch cannot be entered on this guest (no PM), so natural-world reproduction still requires physical port-multiplier hardware.
No comments yet.