DF-4335 — twa CL get/set_param poll-timeout cleanup-contract violation

File: sys/dev/raid/twa/tw_cl_io.c Class: memcorrupt (request-queue corruption → cross-thread request reuse) Severity: Medium — hardware-gated (3ware 9000-series controller), corruption-grade mechanism Status: kernel path NOT runnable on the audit guest (no twa hardware); mechanism reproduced deterministically at logic level with the driver's own queue macros.

What

tw_cli_get_param() and tw_cli_set_param() (poll mode, callback == TW_CL_NULL) submit an internal firmware request via tw_cli_submit_and_poll_request(). On poll timeout that helper enforces an explicit ownership contract (tw_cl_io.c:999-1006):

"If a busy request timed out, the reset will take care of freeing resources. If a pending request timed out, we will free resources for that request, right here … So, the caller is expected to NOT cleanup when TW_OSL_ETIMEDOUT is returned."

Both callers violate it: their out: label unconditionally re-queues the request (tw_cli_req_q_insert_tail(req, TW_CLI_FREE_Q)) whatever the error:

• tw_cli_get_param — submit at tw_cl_io.c:781-784, re-queue at 812-813
• tw_cli_set_param — submit at tw_cl_io.c:899-902, re-queue at 929-930

Combined with the helper's own cleanup:

• PENDING timeout — helper already inserted the request into FREE_Q at tw_cl_io.c:1029 → the caller inserts it a second time.
• BUSY timeout — the request is still linked in BUSY_Q and owned by the firmware; the caller links the same node into FREE_Q → dual queue membership while in flight.

Every insert/remove is individually gen_lock-protected (tw_cl.h:270-313), so the second insert is not rejected — it is performed. Consequences, in escalating order:

1. The doubly-inserted node's next points at itself → the FREE queue is no longer walkable; tw_cli_req_q_remove_head() called from the MPSAFE interrupt path spins forever while holding the gen_lock spinlock → system-wide wedge of every CPU touching any CL request queue.
2. Once the victim reaches the head, every remove_head returns the same struct tw_cli_req_context → two owners build commands in one context (tw_cli_get_request() resets flags but cannot stop the second handout).
3. For the BUSY-timeout variant the firmware still owns the DMA of req->cmd_pkt while a new owner rewrites it; the late firmware response then completes the wrong request, and tw_cli_complete_io() (tw_cl_intr.c:407-416) copies 1024 bytes of firmware response into req_pkt->gen_req_pkt.pt_req.cmd_pkt of whichever OSL request last owned the context — cross-request kernel-memory corruption.

Siblings already filed (same class, different sites): DF-1814 (tw_cli_init_connection, tw_cl_init.c), DF-4274 (OSL deferred-busdma path).

Trigger preconditions

• 3ware 9000/9550SX/9650SE controller present (twa(4) attached).
• An internal poll-mode request must time out (60 s):
• at attach (tw_cli_notify_ctlr_info → get_param, tw_cl_misc.c:671-680) with a wedged/slow firmware, or
• during a controller reset (tw_cl_reset_ctlr → tw_cli_soft_reset → tw_cli_drain_aen_queue/SYNC_TIME → set_param, interrupts down).
• No unprivileged reachability: /dev/twaN is 0600 root:operator (tw_osl_freebsd.c:405-406); the attach/reset paths need no ioctl at all — a wedged firmware alone is enough.

How to reproduce (logic model — no hardware needed)

cd findings/poc/DF-4335
./build.sh          # cc -O2 -Wall -o poc_dblinsert poc_dblinsert.c
./run.sh            # ./poc_dblinsert

Expected output (observed on the audit guest, DragonFly 6.5-DEVELOPMENT x86_64, cc 8.3):

Case 1 (PENDING timeout): FREE_Q walk=INFINITE CYCLE (queue unwalkable; in-kernel this spins while holding the gen_lock spinlock)  <-- BUG
  consumers received 5 request(s); victim handed out 2 time(s) -> SAME REQUEST CONTEXT ISSUED TO TWO OWNERS  <-- BUG
Case 2 (BUSY timeout):   victim on BUSY_Q=1 on FREE_Q=1 -> REQUEST SIMULTANEOUSLY IN BUSY_Q AND FREE_Q  <-- BUG
Case 3 (PENDING, fixed): FREE_Q walk=4 victim_on_free=1 (expect 4 / 1, no cycle)
Case 3 (BUSY, fixed):    victim on BUSY_Q=1 on FREE_Q=0 (expect 1 / 0; reset-drain hook requeues once later)

Cases 1-2 model the shipping code (corruption); Case 3 models the call sequence after fix.diff (clean single membership in both timeout cases).

The reproducer uses the driver's exact TW_CL_Q_* list macros copied from tw_cl_share.h and the exact call sequence of tw_cl_get_param's error path vs tw_cli_submit_and_poll_request's timeout tail. It is a faithful model of the queue-state machine, not of the firmware interaction; the kernel path itself cannot be exercised on a guest without the PCI hardware.

Fix

See fix.diff — callers must not re-queue on TW_OSL_ETIMEDOUT (PENDING case is already cleaned by the helper), and the BUSY case needs a completion hook so the imminent reset drain re-queues the request exactly once (the documented contract is otherwise unimplemented for poll-mode requests, which have no tw_cli_callback).

DF-4335 — VERDICT

Status: untested (kernel path not runnable — no twa(4) hardware on the audit guest; /dev/twa* absent, no PCI vendor 0x13c1 device present). Mechanism: reproduced deterministically at logic level (poc_dblinsert, run on the guest, run.log). Impact class of the defect: request-queue corruption → cross-thread request-context reuse → kernel memory corruption / hard lockup; realistic manifestation on real hardware is panic/wedge (DoS-grade) with a corruption path that cannot be ruled out. Confidence: certain for the code defect (single-threaded, deterministic, no race required — only a 60 s poll timeout of an internal request).

Root cause, line by line

1. tw_cli_get_param() builds an internal request, poll-submits it: - sys/dev/raid/twa/tw_cl_io.c:781-784 c error = tw_cli_submit_and_poll_request(req, TW_CLI_REQUEST_TIMEOUT_PERIOD); if (error) goto out;
2. tw_cli_submit_and_poll_request() on timeout, PENDING case, cleans up the request itself: - sys/dev/raid/twa/tw_cl_io.c:1014-1030 — removes from PENDING queue, tw_cli_req_q_insert_tail(req, TW_CLI_FREE_Q) at 1029, returns TW_OSL_ETIMEDOUT. - The ownership contract is written out at 999-1006: "the caller is expected to NOT cleanup when TW_OSL_ETIMEDOUT is returned"; for the BUSY case "the reset will take care of freeing resources".
3. The caller ignores the contract and cleans up anyway: - tw_cl_io.c:804-814 (get_param out:) — tw_cli_req_q_insert_tail(req, TW_CLI_FREE_Q) at 812-813, unconditional on the error value. - tw_cl_io.c:921-931 (set_param out:) — same, at 929-930.

Result by case:

• PENDING timeout: node inserted into FREE_Q twice. The second TW_CL_Q_INSERT_TAIL (tw_cl_share.h:288-296) sets item->next = head and item->prev = head->prev where head->prev is the same node (it is already the tail) → v->next = v: the circular list now has a self-loop. Verified with the driver's own macros: walking the queue never terminates (run.log Case 1) and once the node reaches the head, every tw_cli_req_q_remove_head() returns it again (Case 1, "handed out 2 time(s)"). Queue ops hold ctlr->gen_lock — a spinlock (tw_cl.h:274, spin_init at tw_osl_freebsd.c:307) — so in-kernel the walk spins with preemption disabled.
• BUSY timeout: the request is still linked in BUSY_Q and owned by the firmware; the caller's insert links the same node into FREE_Q → simultaneous membership (run.log Case 2). The request is then handed out again by tw_cli_get_request() (tw_cl_misc.c:569-606; it resets state/flags but nothing prevents the second handout), its cmd_pkt DMA buffer is rebuilt while the firmware still owns the original command, and the late response for the old request_id completes whichever request currently sits in that context — including the 1024-byte copyback at tw_cl_intr.c:407-410 into the wrong OSL request's buffer.

Why the reset does not save the PENDING case either

For a poll-mode (callback == NULL) request, req->tw_cli_callback is NULL, so tw_cli_drain_busy_queue (tw_cl_misc.c:140-170) sets error_code and calls no callback — the documented "reset will take care of freeing resources" is unimplemented for poll-mode requests. The request would leak in no queue at all were the caller's unconditional insert removed; hence the fix adds a requeue callback for the BUSY-timeout case instead of merely deleting the caller's insert (see fix.diff).

Why untested (not not_reproduced / not_testable)

The defect is in the twa(4) common layer; reaching it requires a 3ware 9000-series PCI controller (attach path) or a controller reset (wedged firmware). The audit QEMU guest has no such device: ls /dev/twa* → No such file or directory; pciconf -l | grep -c 13c1 → 0 (build.log). There is no software-only entry point: /dev/twaN exists only when the driver attached to real hardware. The logic-level reproduction (poc_dblinsert) was built and run on the guest itself to demonstrate the queue-state machine corruption with the driver's own list macros.

Exploit chain (hardware present, outline)

operator/root or wedged-firmware-at-boot → internal get/set_param request times out after 60 s with PENDING (or BUSY) state → FREE_Q self-loop / dual-queue membership → (a) wedge: any CPU taking gen_lock spins forever (hard DoS), or (b) same request context handed to two owners → cmd_pkt rewritten under live firmware DMA + late response completing the wrong request → 1024-byte firmware-controlled copyback into a foreign OSL request buffer (kernel heap corruption). Deterministic escalation beyond panic is hardware- and firmware-dependent; not demonstrable on this guest.

Fix validation

Not possible on this guest (no twa hardware): fix_status: not_testable, fix_baseline_reproduced: 0, fix_patched_reproduced: 0. fix.diff was authored against the read-only sys/ tree after verification; it was never applied there. The logic model can be re-pointed at the patched call sequence to show both cases return to single-queue membership.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.