DF-4509 — twa(4) tw_cli_drain_aen_queue() FREE_Q double-insert on ETIMEDOUT (request left dual-linked in BUSY_Q/PENDING_Q + FREE_Q)

• File: sys/dev/raid/twa/tw_cl_misc.c:309-386 (out: label :379-384, ETIMEDOUT break :353-356)
• Class: queue-discipline / latent use-after-free (memcorrupt bucket)
• Siblings: DF-4335 (get/set_param requeue), DF-4274 (OSL passthru), DF-1814 (init_connection)

What it is

When the 3ware firmware stops completing commands, the controller-reset path calls tw_cli_drain_aen_queue(), which submits a REQUEST_SENSE by internal request and polls for TW_CLI_REQUEST_TIMEOUT_PERIOD. The request is at that moment linked into TW_CLI_BUSY_Q (tw_cl_io.c:219) or TW_CLI_PENDING_Q (tw_cl_io.c:204). If the poll times out, the function breaks to its out: label and unconditionally inserts the same request into TW_CLI_FREE_Q — without unlinking it from the queue it still occupies. The list macros (TW_CL_Q_INSERT_TAIL, tw_cl_share.h:284-289) overwrite the item's next/prev, cross-linking the two rings: the request can subsequently be handed out by tw_cli_get_request() while the firmware still owns it, its internal_req_data DMA buffer is re-assigned to a second command, and the later firmware completion / reset drain completes the reused request into the wrong owner's callback. Same corruption class as DF-4335/DF-4274, third distinct site.

Reproduce (on twa-equipped hardware)

1. ./build.sh && ./run.sh — run.sh is a presence prober; it lists /dev/twa*.
2. Real trigger: hang the controller firmware (fatal drive-slot fault, or devctl-detach the device MMIO region while I/O is in flight) so the watchdog/passthru-timeout reset path fires; the reset's post-reset AEN drain then hits the 60s REQUEST_SENSE timeout and the request is double-queued. Expected observation: subsequent kldunload/reset hangs, duplicated completions in the trace buffer, or a panic in the request queue walk.

On the audit guest

twa_probe reports no /dev/twa* nodes ... controller ABSENT. The QEMU guest has no 3ware 9000-series PCI device; the driver never attaches. The defect is proven by code inspection in VERDICT.md (deterministic queue-state analysis); verdict: inconclusive (hardware absent), not a false positive.

Fix

fix.diff — unlink the request from BUSY_Q/PENDING_Q at the out: label before inserting it into FREE_Q.

DF-4509 — twa(4) tw_cli_drain_aen_queue() re-queues a still-busy request into FREE_Q on ETIMEDOUT → dual queue membership → request reuse / list corruption (latent UAF)

Status: inconclusive on the audit guest (no twa hardware). The defect is proven deterministically by code inspection below; it cannot be exercised on a machine without a 3ware 9000-series controller, and even with one it requires the controller firmware to stop completing commands (the precise failure mode that triggers the reset path in the first place).

Root cause (line-accurate)

tw_cli_drain_aen_queue() — sys/dev/raid/twa/tw_cl_misc.c:309-386:

319:  if ((req = tw_cli_get_request(ctlr)) == TW_CL_NULL) { ... }   // req UNLINKED from FREE_Q
325:  req->flags |= TW_CLI_REQ_FLAGS_INTERNAL;
326:  req->tw_cli_callback = TW_CL_NULL;
327:  if ((error = tw_cli_send_scsi_cmd(req, 0x03 /* REQUEST_SENSE */))) ... break;
     // tw_cl_io.c:1304-1308: internal_req_busy=TRUE; req->data=internal buffer
     // tw_cl_io.c:1340 -> tw_cli_submit_cmd(req):
     //   queue full   -> tw_cl_io.c:199-204: state=PENDING; insert_tail(req, PENDING_Q)
     //   submitted ok -> tw_cl_io.c:218-219: state=BUSY;    insert_tail(req, BUSY_Q)
337:  do { ... poll tw_cli_process_resp_intr(req->ctlr) ... } while (time <= end_time);
353:  if (req->state != TW_CLI_REQ_STATE_COMPLETE) {
354:      error = TW_OSL_ETIMEDOUT;      // <-- firmware never completed the sense
355:      break;                          // req STILL LINKED in BUSY_Q or PENDING_Q
356:  }
...
379: out:
380:  if (req) {
381:      if (req->data)
382:          ctlr->internal_req_busy = TW_CL_FALSE;   // internal DMA buffer also released
383:      tw_cli_req_q_insert_tail(req, TW_CLI_FREE_Q); // <-- SECOND membership, no unlink
384:  }

The queue-insert macros (TW_CL_Q_INSERT_TAIL, sys/dev/raid/twa/tw_cl_share.h:284-289) blindly overwrite item->next/item->prev and link the item into the new ring. They do not unlink the item from any list it is still on. After the ETIMEDOUT break, req is simultaneously:

• linked in TW_CLI_BUSY_Q (via tw_cl_io.c:219) or TW_CLI_PENDING_Q (via tw_cl_io.c:204), and
• linked in TW_CLI_FREE_Q (via tw_cl_misc.c:383),

with BUSY_Q/PENDING_Q's neighbours still pointing at req while req's own pointers now describe FREE_Q's ring — the two rings are cross-linked.

Consequence chain

1. tw_cli_get_request() (tw_cl_misc.c:578) pops req from FREE_Q and hands it to a new owner (e.g. the next passthru ioctl, AEN fetch, or param request), zeroing req->flags, req->orig_req, req->tw_cli_callback (tw_cl_misc.c:583-591) while req is still recorded as in-flight with the firmware.
2. internal_req_busy was cleared at tw_cl_misc.c:382 although the original REQUEST_SENSE still owns ctlr->internal_req_data (assigned at tw_cl_io.c:1305); the next internal command reuses the same DMA buffer for a different command while the firmware may still write the old one.
3. When the hung firmware eventually completes (or a subsequent reset drains queues): tw_cli_process_resp_intr (tw_cl_intr.c:238-260) looks the request up by firmware id, finds state == BUSY (it was re-submitted by the new owner), removes it from BUSY_Q, and completes it into the new owner's callback context; or tw_cli_drain_busy_queue (tw_cl_misc.c:140-168) completes the same request again for the old context. One struct tw_cli_req_context with two owners ⇒ crossed completion callbacks, double insertion into FREE_Q from both contexts (the exact corruption class of DF-4335/DF-4274), and eventually the request-ring macros writing through stale neighbour pointers once the duplicate is unlinked twice (TW_CL_Q_REMOVE_ITEM, tw_cl_share.h:293-296) — latent memory corruption of the driver's static request array and a request reused while its 2048-byte cmd_pkt is under live firmware DMA (tw_cl_init.c:341-346 maps each request to its own command packet).

Why inconclusive on the guest

$ ls /dev/twa*          -> No such file or directory
$ pciconf -l            -> no 0x13C1/0x1002-class 3ware 9xxx device

The audit QEMU guest has no twa controller; the driver never attaches, so the reset/AEN machinery never executes. There is no unprivileged (or even root-usable) software trigger short of the hardware hanging.

Family / siblings (not duplicates)

• DF-4335 — same corruption class, different sites: tw_cli_get_param / tw_cli_set_param (tw_cl_io.c) re-queue after ETIMEDOUT during normal operation (every AEN sync-time, every management ioctl).
• DF-4274 — OSL passthru deferred-busdma error path returns a request to FREE_Q twice (tw_osl_freebsd.c).
• DF-1814 — tw_cli_init_connection double-insert on submit_and_poll failure.
• This finding (DF-4509) — the reset-path AEN drain in the misc layer, reachable exactly when the controller is being recovered from a hang.

Recommended fix

Unlink the request from whichever queue it still occupies before returning it to FREE_Q at the out: label — see fix.diff.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.