DF-4060 — amr_std_poll_command timeout leaks the slot as permanently busy

Status: REPRODUCED (guaranteed follow-on panic via production amr_start; completion callback + status stores into a free-listed command; rescan_drives livelock precondition) — production-handler KLD dispatch rig (shared with DF-4059), no MegaRAID hardware needed. Fix validated on a rebuilt kernel. See VERDICT.md.

What runs

Same rig as DF-4059 (dfamr_rig.c, root-only /dev/dfamr0): fabricated post-attach softc with real bus_dma, production command clusters, production amr_std_poll_command / amr_start / amr_alloccmd / amr_releasecmd / amr_done dispatched from the shipped kernel image. The stall condition is modeled exactly: the virtual card accepts the posted command and never raises a completion.

Build

findings/poc/DF-4060/build.sh        # same rig as DF-4059

Run (baseline)

./dfbsd-qemu/vm.sh run_root 'cd /root/rig && kldload ./dfamr_rig.ko && \
    ./dfamr_ctl prime && ./dfamr_ctl offs && ./dfamr_ctl magic && \
    ./dfamr_ctl tmo 0 && ./dfamr_ctl late'
# deterministic panic finale (last, guest dies):
./dfbsd-qemu/vm.sh run_root 'cd /root/rig && ./dfamr_ctl tmo 1'

Expected results

• baseline tmo 0: poll_err=EIO after ~103 ticks; after timeout BUSY still set, busycmd[slot]==ac, busyslots==1; after the caller's releasecmd the still-busy command is the freelist head; 100x amr_done never drains busyslots (rescan_drives amr.c:460 livelock precondition); the next amr_alloccmd returns the same command
• baseline tmo 1: panic: amr: slot 2 busy? at amr_start+0x18b (amr.c:1742-1743) — guest dead
• baseline late: production amr_done completes the stale id into the free-listed command: status 0x77 stored, completion callback fired while the command is still on amr_freecmds (two owners)
• patched (fix.diff): timeout detaches the slot (busyslots=0, busycmd cleared, BUSY cleared); tmo 1 returns start_err=0, no panic; late completion tolerated as "bad slot 2 completed"; guest alive

Fix

fix.diff (shared with the DF-4059 pack): detach the slot under amr_hw_lock on the amr_std_poll_command timeout path (guarded on busycmd[slot] == ac) and clear AMR_CMD_BUSY.

DF-4060 — VERDICT

REPRODUCED (guaranteed panic from the poisoned slot + completion of a free-listed command + livelock precondition; fix validated on a rebuilt kernel).

Same production-handler dispatch rig as DF-4059 (dfamr_rig.c; shared pack artifacts, see findings/poc/DF-4059/). Production amr_std_poll_command is dispatched from the shipped kernel image (nm t amr_std_poll_command @ 0xffffffff80345550, prologue-verified) over a fabricated post-attach softc with real bus_dma and a command cluster built by the verbatim amr_alloccmd_cluster; the "card" is modeled by the memory register file + rig_get_work wrapper (production amr_std_get_work + card-side interrupt drop) so the poll loop's 1001 amr_done calls exercise production code. The stall condition of the finding is modeled exactly: the virtual card consumes the posted command and never raises a completion.

Reproduction 1 — the timeout poisons the slot; next start panics (bootA)

Production sequence (mirrors amr_flush amr.c:1086-1093): amr_alloccmd → AMR_CMD_PRIORITY|AMR_CMD_DATAOUT, mb_command = AMR_CMD_FLUSH → production amr_std_poll_command(ac):

[4060TMO] poll_err=5 ticks=103          (EIO after ~1.03 s production timeout loop)
[4060TMO] after timeout: BUSY_still_set=1 busycmd_holds_ac=1 busyslots=1
[4060TMO] after releasecmd: freelist_head_is_ac=1; busyslots after 100x amr_done=1 (rescan_drives livelock precondition)
[4060TMO] recycled_same_cmd=1

• amr.c:1356-1362: on timeout the slot stays busy — AMR_CMD_BUSY still set, amr_busycmd[slot] == ac, amr_busyslots == 1 (the "XXX the slot is now marked permanently busy" comment is accurate).
• Every caller releases unconditionally on error (amr_enquiry:1058-1061, amr_flush:1097-1100, amr_support_ext_cdb:1140-1143, amr_dump_blocks:2380-2382) — rig does the same amr_releasecmd: the still-busy command becomes the free-list head.
• 100 further production amr_done calls never drain amr_busyslots (stays 1): the while (sc->amr_busyslots) loop of amr_rescan_drives (amr.c:460-463, reachable from the AMR_IO_CONFIG delete/create ioctl path amr.c:896-897 / amr.c:676) spins forever — livelock precondition proven; the loop body is device_printf + amr_done, i.e. an unkillable console-flood spin on a real box.
• Next amr_alloccmd hands back the same command; production amr_start then hits its guard:

[amrrig] DF-4060 FINALE: production amr_start on the recycled command (slot 2 still busy)
panic: amr: slot 2 busy?
amr_start() at amr_start+0x18b     (amr.c:1742-1743)

(bootA_serial.log, panic_4060.txt; guest dead in DDB.) Deterministic: any driver activity after one timed-out polled command panics the kernel.

Reproduction 2 — late completion completes the recycled command (bootA)

After timeout + release (command on the free list), the card finally completes the stale id. To capture the stores before amr_startio re-allocates the very same command off the free-list head (its amr_alloccmd:1895 field-clear wipes ac_status — itself part of the confusion), the rig arms ac_complete on the free-listed command first, a stand-in for amr_completeio, i.e. exactly the finding's "command recycled for a new bio" case:

[amrrig] DF-4060LATE: probe fired=1 on cmd 0xfffff801187c41e0:
    completion status 0x77 stored, BUSY at callback 0, ON FREE LIST at callback 1; acks=1
[4060LATE] poll_err=5; status 0x0->0x77 busy 1->0 still_on_freelist=1 busyslots=0

Production amr_done → amr_freeslot + ac->ac_status = 0x77 (amr.c:1795-1798) + amr_complete → completion callback executed on a command that simultaneously sits on amr_freecmds (two owners). On a live system where the recycled command already carries a new bio, this is the finding's "wakeup/status stores into a live unrelated command" / double-completion of the new owner's request (the race window is between amr_releasecmd and the next amr_start; the single-threaded rig shows the mechanism deterministically via the callback).

Threat model

Trigger precondition is a controller that stalls a polled command ~1 s (broken/faulty card, or the same malicious-device class as DF-4059). The follow-on panic needs no further malfunction: the next driver-issued command (ioctl, rescan, shutdown flush amr_pci_shutdown:400, dump) hits the poisoned slot. Root-only surface (/dev/amr0 0600 + card presence), so severity per the filing (Medium) stands; the class is kernel memory/state corruption with a guaranteed crash and a racy double-completion/UAF facet.

Fix validation (patched kernel #1, nativekernel gcc 8.3, rc=0)

fix.diff (shared with DF-4059's pack) detaches the slot on the amr_std_poll_command timeout path under amr_hw_lock (guarded on busycmd[slot] == ac so a racing real completion is harmless) and clears AMR_CMD_BUSY. On the rebuilt kernel (run_patched.sh / fix_run.log):

• tmo (with the former panic finale enabled): poll_err=5, busyslots=0 after timeout, busycmd cleared, BUSY cleared, recycled_same_cmd=1 and amr_start on the recycled command returns 0 — no panic; guest alive (uptime shown);
• late: the stale completion now finds busycmd[idx] == NULL → bad slot 2 completed tolerated, probe not fired, no completion stores into the free-listed command;
• rescan precondition gone with the counter (busyslots reaches 0, so while (sc->amr_busyslots) exits immediately).

Baseline panic/double-ownership behaviors gone; normal completion of real ids unaffected (the tmo run's newly-started command completes cleanly through the patched loop).

Adjacent, not re-verified here

The quartz poll paths have sibling leak shapes (amr_quartz_poll_command bus_dmamap_load failure at amr.c:1421-1425 and the "adapter is busy" return at amr.c:1448-1456 return with the slot still marked busy by amr_start); the fix's guard pattern applies to them identically, but they are out of the verified scope of this finding (filed scope: amr_std_poll_command).

Artifacts

Shared rig with DF-4059; this pack: dfamr_rig.c, dfamr_ctl.c, Makefile, build.sh, plus from the DF-4059 pack: bootA_serial.log (full baseline sequence incl. both 4060 reproductions and the finale panic), panic_4060.txt, fix.diff, fix_build.log, fix_run.log.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.