DF-4918 — drain_workqueue() sleeps holding worker_lock → permanent kernel-thread/workqueue deadlock

sys/dev/drm/linux_workqueue.c:276-281:

lockmgr(&worker->worker_lock, LK_EXCLUSIVE);
while (!STAILQ_EMPTY(&worker->ws_list_head)) {
        tsleep(&drain_workqueue, 0, "wkdrain", 1);   /* plain tsleep */
}
lockmgr(&worker->worker_lock, LK_RELEASE);

Plain tsleep() does not release held locks (only lksleep() does). The worker thread needs that very lock to dequeue and run the remaining works (process_all_work() re-acquires at linux_workqueue.c:91), so once drain_workqueue() acquires the lock of a worker whose queue is non-empty:

• the drainer loops forever, one tick at a time, holding the lock;
• the worker blocks forever in lockmgr() (wchan lwq);
• every later queue_work() on the same workqueue blocks forever at linux_workqueue.c:134 — cascading wedge of arbitrary kernel threads.

Distinct from DF-1977 ("drain does not wait for in-flight works" — returns too early): here drain never returns and permanently wedges the wq. Root cause is different (sleep-with-lock-held vs missing in-flight wait).

Reached in-tree from destroy_workqueue() (linux_workqueue.c:246) — every DRM driver teardown — and from direct drain_workqueue() callers, with any works still pending on the queue at drain time.

Build (guest, root)

kldload drm                # workqueue shim lives in drm.ko
cd /tmp/df4918 && make

Run

kldload /tmp/df4918/df4918.ko
sysctl hw.df4918_test=1    # ~16s; prints per-second state
sysctl hw.df4918_status    # re-print frozen state

Expected

Unpatched (baseline run.log / ps output):

[DF4918] t+ 3s..14s: drain_done=0 filler_done=0 slow_done=1 probe_done=1
[DF4918] DEADLOCK CONFIRMED: drain_workqueue() never returned, ...
ps -ax -o wchan:   lwq      B1  df4918wq      <- worker blocked on worker_lock
                  wkdrain  B2  df4918drain    <- drainer asleep HOLDING it

Patched (run_fixed.log): drain_done=1 filler_done=1 within seconds, guest healthy.

The wedge is permanent (threads park; module refuses unload) — reset the guest afterwards: vm.sh reset with-src.

DF-4918 VERDICT — REPRODUCED (deterministic deadlock, DoS)

What was run

Kernel module df4918.ko on the audit guest (DragonFly 6.5-DEVELOPMENT, SMP 6 CPU, drm.ko stock and patched), root-loaded, sysctl-triggered:

1. own workqueue alloc_workqueue("df4918wq", WQ_UNBOUND, 1) → single worker (workers[0]), exactly like create_singlethread_workqueue.
2. queue_work(slow_fn) — worker dequeues it and sleeps 3 s inside the func (process_all_work released worker_lock for the func body, linux_workqueue.c:88).
3. queue_work(filler_fn) — sits in ws_list_head while slow_fn runs: queue non-empty, lock free.
4. drain thread calls drain_workqueue(wq) → is_draining=true, then lockmgr(EXCLUSIVE) on workers[0] (succeeds — the lock is free), !STAILQ_EMPTY → tsleep(&drain_workqueue, 0, "wkdrain", 1) with the lock held.
5. probe thread calls queue_work(wq, probe) (returns false immediately — is_draining short-circuits at :126; on a non-draining wq it would block forever on the held lock).

Result (baseline, stock drm.ko)

• slow_done=1 at t+3s (func finished, worker tried to re-acquire the lock at process_all_work:91 → blocked).
• drain_done=0, filler_done=0 for the whole 14 s observation window and on every later sysctl hw.df4918_status poll — permanent.
• ps -ax -o wchan proof: worker df4918wq blocked in lwq (the worker_lock lockmgr wait channel — lockinit name at linux_workqueue.c:220), drainer df4918drain asleep in wkdrain holding that lock. Deadlock closed end-to-end.
• The guest had to be hard-reset afterwards; during one shutdown attempt the wedged threads also hung the shutdown path (guest ended at db>), i.e. the wedge propagates past the wq itself.

100 % deterministic: two independent baseline runs, identical outcome.

Fix validation

fix.diff (generated against the read-only sys/ tree; applied to the guest's /usr/src copy, drm.ko rebuilt):

tsleep(...) → lksleep(&drain_workqueue, &worker->worker_lock, 0, "wkdrain", 1) — releases the lock while sleeping, reacquires on return (the same primitive the file itself uses at :110 and :382).

Patched kernel, identical test: drain_done=1 filler_done=1 slow_done=1 within seconds, "drain completed (no deadlock) — fixed kernel?", guest healthy (run_fixed.log).

Impact

Permanent kernel-thread + workqueue wedge (local DoS class, same class as DF-1976/DF-1977): every destroy_workqueue()/drain_workqueue() with pending works wedges the caller, the worker(s), and (via worker_lock) any queue_work() on that wq — driver unload/reset/suspend paths. No memory-corruption component.

Kernel references

• sys/dev/drm/linux_workqueue.c:265-286 (drain_workqueue: tsleep with worker_lock held at :279)
• sys/dev/drm/linux_workqueue.c:88-91 (process_all_work lock protocol)
• sys/dev/drm/linux_workqueue.c:126-146 (queue_work lockmgr / cascade)
• sys/dev/drm/linux_workqueue.c:220 (lockinit "lwq" — the observed wchan)

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.