DF-2870 — taskqueue_free() terminate lost-wakeup → permanent deadlock

What

taskqueue_thread_loop() (subr_taskqueue.c:617-619) parks in TQ_SLEEP(tq, tq, "tqthr") after taskqueue_run() without re-checking TQ_FLAGS_ACTIVE. taskqueue_terminate() (subr_taskqueue.c:141-147) delivers wakeup(tq) after dropping the queue spinlock; if the worker is executing a task function at that moment (not on the sleep queue), the wakeup is lost. When the func returns, the worker parks unconditionally, the terminator is asleep on tq_threads, and nobody is left to wake either — taskqueue_free() never returns and the worker thread is leaked.

Trigger: taskqueue_free() called while any task of the queue is in flight (dequeued, ta_func executing). Every in-tree taskqueue_free() consumer (wlan ieee80211.c:464, ~15 network/storage drivers) reaches this window when tearing down a busy queue.

Build (on the guest as root)

same out-of-tree KLD pattern: see build.sh

Run

kldload /root/poc/tqdead/tqdead.ko
sleep 15
dmesg | grep tqdead
ps -axH -o stat,wchan,comm | grep -E "tqdead"

Expected output (stock kernel — reproduced)

tqdead: queue 0xfffff8008da44e88 started (0)
tqdead: task in-flight; calling taskqueue_free()...
tqdead: releasing in-flight task func
tqdead: RESULT: DEADLOCK CONFIRMED - taskqueue_free() stuck for 10+ seconds;
        ctrl_td=0xfffff8008df45b80 (wchan 0xfffff8008d560c80), worker parked forever
--- ps ---
B2     taskqueu tqdeadctrl     <- asleep in "taskqueue_terminate" forever
B4     tqthr    tqdead         <- worker parked forever

100% deterministic (no timing race needed — the harness blocks the func until after free() has parked).

NOTE: the wedged module cannot be unloaded; reset the guest afterwards.

Fixed kernel

worker re-checks TQ_FLAGS_ACTIVE before parking → taskqueue_free RETURNED (not reproduced) and the module unloads cleanly (see run.fixed.log).

DF-2870 VERDICT — REPRODUCED (permanent kernel deadlock, 100% deterministic)

One-line

taskqueue_free() called while a task is in-flight deadlocks forever: taskqueue_terminate()'s wakeup(tq) (subr_taskqueue.c:144, delivered after TQ_UNLOCK) is lost because the worker is inside ta_func and not on the sleep queue; when the func returns, taskqueue_thread_loop() parks in TQ_SLEEP(tq, tq) at :619 WITHOUT re-checking TQ_FLAGS_ACTIVE, and the terminator is already asleep on tq_threads (:146) — both sleep forever.

Interleaving (proven on the guest)

• worker: dequeues task (ta_pending=0), releases TQ_LOCK, executes ta_func (harness blocks it on a flag) — not on any sleep queue for ident tq.
• freer: taskqueue_free() → TQ_LOCK, clears TQ_FLAGS_ACTIVE (:154), taskqueue_run(queue,1) returns instantly (queue empty — the in-flight task was already dequeued), taskqueue_terminate(): TQ_UNLOCK → wakeup(tq) → lost (worker not sleeping) → TQ_LOCK → parks in TQ_SLEEP(pp).
• harness releases the func; worker: wakeup(task), TQ_LOCK, run loop sees empty queue, parks in TQ_SLEEP(tq, tq) — without re-checking ACTIVE.
• Both threads sleep with no timeout; nobody can wake them: enqueues are rejected (EPIPE, :199) so taskqueue_thread_enqueue never fires, and the only wakeup(tq) source is the parked terminator.

Evidence

• run.log: tqdead: RESULT: DEADLOCK CONFIRMED - taskqueue_free() stuck for 10+ seconds and ps -axH showing tqdeadctrl in wchan "taskqueue_terminate" (truncated "taskqueu") and the worker in "tqthr", both permanently.
• The escape case (worker parked before free) is the properly-drained case; the deadlock hits whenever a task is in-flight at free time — the exact situation free()'s run+terminate machinery exists to handle (subr_taskqueue.c:153-156).

In-tree reachability

Every taskqueue_free() consumer (wlan ieee80211.c:464; ~15 network and storage drivers in the callers list) that frees a still-active queue wedges the calling thread forever — a detach-path hang under load (local DoS; also remotely-triggerable if the detach is event-driven, e.g. device removal during traffic).

Fix validation

fix.diff: worker re-checks TQ_FLAGS_ACTIVE (under the still-held TQ_LOCK) between taskqueue_run() and TQ_SLEEP, exiting instead of parking. On the fixed kernel the harness prints taskqueue_free RETURNED (not reproduced) and the module unloads cleanly. See run.fixed.log.