DF-5058 — bios submitted to a never-completing mlx controller park on

mlx_bioq forever: unkillable D-state issuers + unbounded bioq growth

file: sys/dev/raid/mlx/mlx_disk.c (submit path; parking logic in sys/dev/raid/mlx/mlx.c) bucket: base:dev (DF-5026 / DF-5048 family — the mlx instance, the last LD sibling) severity: Low (trigger = controller hardware fault: accepts commands then stops completing, or wedges with slots held)

Reproduce

Not runnable on this guest: no Mylex PCI hardware (env.txt), driver never attaches. Family twin DF-5048 (ida) was accepted with the same hardware-gated pack.

Expected behavior on real hardware

1. Saturate /dev/mlxd0 with asynchronous I/O (e.g. several parallel dd/vinum readers).
2. Cause the controller to stop delivering completions after accepting commands (firmware hang / surprise fault while BUSY).
3. After the 256 slots fill, every further bio parks on sc->mlx_bioq (mlx.c:1739-1742). No timeout exists anywhere (mc_timeout is set at mlx.c:2086 and explicitly "not currently used"); mlx_periodic only polls mlx_done every second (mlx.c:1040) and never fails a stale command.
4. Every issuer sleeps in biowait — unkillable D-state processes; the bioq grows without bound (kernel memory exhaustion).

Fix

fix.diff: stall watchdog inside the existing 1 Hz mlx_periodic callout — if parked bufs exist and mlx_done completed nothing for 60 consecutive seconds, fail every queued bio with ETIMEDOUT and fail new arrivals until completions resume.

DF-5058 VERDICT — parked bios / no completion path on a wedged mlx controller

status: untested (hardware-gated) | impact: none demonstrated (local DoS class) | confidence: certain (code trace)

Reproduced?

No. Requires a physical Mylex DAC960 controller that stops completing I/O. Guest evidence (env.txt): no 0x1069 PCI function, no 0x1011:1065 function, /dev/mlx*//dev/mlxd* absent, driver compiled into the running kernel (X86_64_GENERIC:134). Untested = hardware absent — same basis the family accepted for DF-5048.

Why the finding is real (trace)

Completion-flow enumeration — there is no biodone for bios that enter mlx_submit_bio other than mlxd_intr via mlx_completeio:

1. mlxd_strategy (mlx_disk.c:156-190) local error paths all biodone (mlx_disk.c:166-170, 173-177, 183-189) — those are fine (this is the DF-5012 class, already clean here).
2. The submit path (mlx_disk.c:179-181) → mlx_submit_bio (mlx.c:683-693): bioqdisksort(&sc->mlx_bioq, bio) + mlx_startio.
3. mlx_startio (mlx.c:1713-1805): pops a bio only when mlx_alloccmd succeeds and mlx_getslot returns 0; out of command slots (mlx.c:1739-1742, limit - 4 reached) → break, bio stays parked on mlx_bioq. Commands whose mlx_start fails outright are completed with WEDGED (mlx.c:1796-1800 → mlx_completeio → mlxd_intr) — the fast wedge path is handled; the parked path below is not.
4. The ONLY restart for parked bios is mlx_done → mlx_startio (mlx.c:2160-2161), reached from the hardware interrupt (mlx.c:668-676) or the 1 Hz mlx_periodic (mlx.c:1040) — both rely on the controller eventually delivering completions.
5. No timeout exists anywhere: mc_timeout = time_uptime + 60 is set in mlx_start (mlx.c:2085-2086) under the comment "XXX not currently used"; nothing ever reads it. mlx_complete (mlx.c:2173-2223) only processes mc_status != MLX_STATUS_BUSY; the periodic enquiry/rebuild polls (mlx.c:977-1044) never fail stale commands.
6. Therefore: controller accepts commands then never completes them ⇒ all slots (up to me_max_commands, mlx.c:1961-1966) stay BUSY forever ⇒ mlx_getslot EBUSY ⇒ every subsequent bio parks on mlx_bioq indefinitely ⇒ biowait sleepers are unkillable D-states (local DoS of every process touching the array; if root/swap live on mlxd the box is dead) and mlx_bioq grows unbounded → kernel memory exhaustion. mlxd_strategy itself has no timeout/cancel path (mlx_disk.c:172-181).

Trigger requires a real controller fault (firmware hang after accept, surprise removal while BUSY, channel stuck paused) — hence Low, matching the DF-5048 (ida) precedent; DF-5026 (ips) was Medium because an adapter-offline config state parked bios deterministically.

Not a duplicate

• DF-3951 (mlx.c): wedge path of mlx_start leaks mlx_busycmds / leaves the DMA map loaded — that is the fast-fail path; this finding is the parked-bioq/issuer-hang path (bios never dequeued at all).
• DF-3950 (mlx.c): mlx_wait_command infinite 1 s-sleep loop for synchronous internal commands (mlxctl rebuild/check) — not disk bios.
• DF-2074 / DF-5057: different mechanisms (bio_driver_info staleness, detach lifetime).
• DF-5026 / DF-5048: the same defect in ips/ida; this is the mlx instance.

Fix assessment

fix.diff adds a stall watchdog to the existing 1 Hz mlx_periodic callout (no new callout needed): a stall-tick counter in struct mlx_softc (mlxvar.h) is reset whenever mlx_done completes anything (mlx.c:1040) and incremented while mlx_waitbufs > 0; 60 consecutive no-progress seconds ⇒ drain mlx_bioq failing every bio with ETIMEDOUT+B_ERROR and fail new arrivals in mlx_submit_bio until completions resume. 60 s is far beyond any healthy controller's worst-case completion latency under load (load always produces completions, which reset the counter). Commands already stuck in hardware keep their slots (fail-stop: the array stays EIO until reset) — noted as the conservative choice. Cannot be build/boot-validated (no hardware); diff is git-apply-clean against sys/.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.