DF-4701 — twa(4): tw_cl_shutdown_ctlr leaves a half-shut-down, still-attached controller when the firmware handshake fails; no ctlr->active gate on the I/O entry paths

• File: sys/dev/raid/twa/tw_cl_init.c (pass 2)
• Severity: Low · Class: improper shutdown state machine / missing liveness gate (CWE-667 / CWE-362)
• PoC status: untested — requires a physical 3ware 9000-series PCI/PCIe controller (vendor 0x13C1, device 0x1002–0x1005). The audit QEMU guest has no such device (see env.txt): twa never attaches there, so neither the baseline nor a patched kernel can exercise the path on the guest.

What the bug is

twa_detach() (tw_osl_freebsd.c:813-815) calls twa_shutdown() and aborts the detach if it returns an error. twa_shutdown() calls tw_cl_shutdown_ctlr(handle, 0) (tw_osl_freebsd.c:853), which:

1. marks the controller inactive (ctlr->active = TW_CL_FALSE, tw_cl_init.c:569),
2. disables interrupts (tw_cl_init.c:571; the OSL already did bus_teardown_intr, tw_osl_freebsd.c:846, and cancelled the watchdog callouts, :849-850),
3. sends the shutdown message via tw_cli_init_connection (tw_cl_init.c:574-576) — on a wedged/hung firmware this times out and returns an error,
4. destroys the CL locks unconditionally, even after the failed handshake (tw_osl_destroy_lock(gen_lock/io_lock), tw_cl_init.c:587-588),
5. returns the handshake error (tw_cl_init.c:591).

The caller then aborts the detach — but the device is still fully attached: the CAM sim is still registered, /dev/twa0 still exists (both are only torn down later in tw_osli_free_resources, tw_osl_freebsd.c:703/777-779, which was skipped).

Nothing re-arms the driver: interrupts stay torn down, the watchdog stays cancelled, ctlr->active stays FALSE — yet the only place in the driver that checks ctlr->active is the interrupt handler (tw_cl_intr.c:81). Neither twa_action()/XPT_SCSI_IO (tw_osl_cam.c:357-393) nor twa_ioctl() → tw_osli_fw_passthru (tw_osl_freebsd.c:147-175, 895+) nor tw_cl_start_io() / tw_cl_fw_passthru() (tw_cl_io.c:70, 261) checks it. Every subsequent I/O request is therefore submitted to a controller the driver has already told to go down.

How to reproduce (needs the hardware)

See trigger.sh. On a machine with a 3ware 9000/9550/9650/9690 card:

1. Wedge or half-wedge the firmware (e.g. tw_cli maintenance operation that hangs the microcontroller, or a controller that is failing) — the handshake in tw_cli_init_connection must time out (TW_CLI_REQUEST_TIMEOUT_PERIOD).
2. As root: kldunload twa (or unbind via devctl). twa_detach → twa_shutdown → tw_cl_shutdown_ctlr errors → detach aborts, device half-dead but attached. Expected marker: tw_cl_init_connection failed / init_connection failed event (tw_cl_init.c:700-704) followed by Failed to shutdown Common Layer/controller (tw_osl_freebsd.c:854-859) and kldunload: kldunload(...) returned 5.
3. As an unprivileged user, issue I/O against any filesystem on the twa volume (ls -lR /twa-volume). Expected: commands stall for the CAM timeout periods and error out; on INVARIANTS/debug kernels the spin-lock state assertions may fire. The volume stays broken until reboot.

A second-order instance of the same defect: twa_shutdown is the device_shutdown handler (tw_osl_freebsd.c:202). CAM issues SYNCHRONIZE CACHE flushes during system shutdown; if twa_shutdown runs first, those XPT_SCSI_IO CCBs are submitted to a controller that was already told to go down with 1 message credit and torn-down interrupts.

Why this is Low

Triggering the broken state requires root (kldunload) plus a controller whose firmware fails the shutdown handshake. What the defect then buys is that the consequences (dead volume, CAM timeout storms, debug-kernel assertions) are sustained by plain unprivileged disk I/O — there is no self-healing path (watchdog cancelled) and no gate refusing new work.

Fix

fix.diff (authored, never applied to sys/): once tw_cl_shutdown_ctlr has torn the CL context down (locks destroyed, active=FALSE) it must not report an error that makes the OSL abort the detach with the device half-dead — the failure is already logged as an event; and the two CL entry points gain an ctlr->active bail-out so a shut-down controller cleanly refuses work instead of submitting to a dead firmware queue.

DF-4701 VERDICT — twa(4) half-shutdown state on firmware-handshake failure

Status: untested (hardware absent from the audit guest) · impact dos · confidence likely (the state machine is fully source-provable; only the trigger needs physical 3ware hardware).

1. Root cause, line by line

twa_detach aborts on shutdown error and leaves the device attached:

• sys/dev/raid/twa/tw_osl_freebsd.c:813-815 c /* Shut the controller down. */ if ((error = twa_shutdown(dev))) goto out; /* <-- device stays attached */

twa_shutdown tears down interrupts and the watchdog before the CL handshake, then propagates the CL error:

• sys/dev/raid/twa/tw_osl_freebsd.c:846 twa_teardown_intr(sc) (bus_teardown_intr)
• sys/dev/raid/twa/tw_osl_freebsd.c:849-850 callout_cancel(watchdog[0/1])
• sys/dev/raid/twa/tw_osl_freebsd.c:853 tw_cl_shutdown_ctlr(&(sc->ctlr_handle), 0)

tw_cl_shutdown_ctlr commits the teardown before it knows whether the firmware handshake will succeed, and destroys the locks regardless of the handshake result:

• sys/dev/raid/twa/tw_cl_init.c:569 ctlr->active = TW_CL_FALSE;
• sys/dev/raid/twa/tw_cl_init.c:571 tw_cli_disable_interrupts(ctlr);
• sys/dev/raid/twa/tw_cl_init.c:574-581 shutdown-message tw_cli_init_connection(...) → on a wedged firmware this polls TW_CLI_REQUEST_TIMEOUT_PERIOD and returns TW_OSL_ETIMEDOUT (tw_cl_io.c:947-1033) — and additionally trips the known DF-1814 FREE_Q double-insert inside init_connection itself.
• sys/dev/raid/twa/tw_cl_init.c:583-588 locks destroyed unconditionally (tw_osl_destroy_lock(gen_lock), tw_osl_destroy_lock(io_lock) = spin_uninit, tw_osl_inline.h:77-80 — a no-op on this tree, sys/sys/spinlock2.h).
• sys/dev/raid/twa/tw_cl_init.c:591 return(error); → detach aborts.

Post-condition when the handshake fails: device attached, /dev/twa0 alive (destroy_dev only in tw_osli_free_resources, tw_osl_freebsd.c:777-779, which was skipped), CAM sim still registered (tw_osli_cam_detach skipped, :703), ctlr->active == FALSE, interrupts torn down, watchdog cancelled, no self-healing.

2. Why subsequent (unprivileged) I/O reaches the dead controller

The only ctlr->active check in the driver is in the ISR:

• sys/dev/raid/twa/tw_cl_intr.c:81 if (ctlr->reset_in_progress || !(ctlr->active)) goto out;

Neither I/O entry path checks it:

• CAM: twa_action XPT_SCSI_IO sys/dev/raid/twa/tw_osl_cam.c:363-393 → tw_osli_execute_scsi → tw_osli_map_request → tw_cl_start_io sys/dev/raid/twa/tw_cl_io.c:70-101 — no active check; requests are queued/submitted to firmware that was told "going down" with TWA_SHUTDOWN_MESSAGE_CREDITS == 1 (tw_cl_fwif.h:137).
• ioctl: twa_ioctl sys/dev/raid/twa/tw_osl_freebsd.c:147-175 → tw_osli_fw_passthru (:895) → tw_cl_fw_passthru (tw_cl_io.c:261) — no check. (/dev/twa0 is 0660 root:operator, tw_osl_freebsd.c:405-407, so this half needs group operator; the CAM path needs no privilege at all.)

Observed effect chain (expected, not exercised): unprivileged read on a twa-backed filesystem → CCB submitted → no completion (interrupts gone / fw down) → CAM peripheral timeouts (twa_timeout sets reset-needed; the watchdog that would run tw_cl_reset_ctlr was cancelled) → repeated CAM retries/timeouts → dead volume until reboot. On INVARIANTS/debug kernels the reused/spin_uninit-ed lock state trips assertions; on production kernels spin_uninit is /* unused */ (sys/sys/spinlock2.h) so the lock itself is benign and the impact is the DoS.

System-shutdown instance of the same ordering defect: twa_shutdown is the device_shutdown method (tw_osl_freebsd.c:202); CAM's shutdown flush (SYNCHRONIZE CACHE) can be issued after twa_shutdown already closed the connection → same submit-to-dead-controller path during reboot.

3. Why not reproduced on the guest

The QEMU guest (see env.txt) has no 3ware controller (pciconf -l shows no 0x13C1:0x1002-0x1005 device), no /dev/twa*, no twa kld, and no drm/i915 either (display is Bochs 0x1234:0x1111). twa_probe (tw_osl_freebsd.c:228-247 via tw_cl_ctlr_supported, tw_cl_init.c:63-73) never matches, so the code path cannot be entered at runtime — neither baseline nor fix-validation is possible on this guest. Not false_positive: the defect is statically provable in the source (above); it is simply hardware-gated.

4. Fix

See fix.diff:

1. tw_cl_shutdown_ctlr: after the CL context is torn down (locks destroyed, active=FALSE), do not return the handshake error — the caller treats non-zero as "device still fully usable" and aborts the detach in the worst possible state. The failure is already logged via tw_cl_create_event.
2. Defense-in-depth: tw_cl_start_io / tw_cl_fw_passthru refuse work when !ctlr->active (EBUSY) so a shut-down controller rejects new I/O cleanly.

Fix not build/boot-tested: single-fix kernel validation requires the controller. fix_status: not_testable, fix_baseline_reproduced: 0, fix_patched_reproduced: 0.

5. Negative result context (what else was checked in tw_cl_init.c pass 2)

• The ETIMEDOUT re-queue class in tw_cli_init_connection out-label is known DF-1814; its siblings in get/set_param (DF-4335) and drain_aen (DF-4509) are also known. Not re-reported.
• tw_cl_get_mem_requirements / tw_cl_init_ctlr size math is consistent with the OSL allocation (tw_osl_freebsd.c:517-531): ctlr ctx + 256 req ctx + 256 event packets; internal_req_data = cmd_pkts + 512B sector — no overflow (tw_cl_init.c:210-223, 289-356). max_simult_reqs capped at 256 (:190-199, 268-277); max_aens is the constant 256 (tw_osl.h:56) so the % max_aens_supported sites can't divide by zero.
• TW_CL_START_CTLR_ONLY jump (tw_cl_init.c:262-266) is dead: OSL sc->flags only ever holds TW_CL_64BIT_ADDRESSES (1<<0) / TW_CL_64BIT_SG_LENGTH (1<<1) (tw_osl_freebsd.c:511-512), no collision with (1<<2) (tw_cl_share.h:86-89).
• tw_cl_get_pci_bar_info inner switch has no default: an invalid bar_type would return success with uninitialized *bar_num/*bar0_offset/*bar_size (tw_cl_init.c:96-146). Only caller passes the constant TW_CL_BAR_TYPE_MEM (tw_osl_freebsd.c:323) → unreachable today; hardening nit, not filed.
• Init-failure unwind leaves lockinit/spin_init-ed locks behind in freed memory (tw_cl_init.c:322-324 vs tw_osl_freebsd.c:400); benign on DragonFly (nothing registered globally, spin_uninit is a no-op).
• Response-ID trust (req_ctxt_buf[GET_RESP_ID(resp)], tw_cl_intr.c:238) is firmware-controlled but masked to 8 bits and the array holds 256 entries (TW_OSLI_MAX_NUM_REQUESTS = TW_CL_MAX_SIMULTANEOUS_REQUESTS = 256, tw_osl.h:53) → no OOB even from a lying firmware.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.