ata_timeout() mid-callback state_mtx window deadlocks the channel against synchronous callout_cancel (or double-completes the request)
Summary
The AUTOLOCK timeout callback drops state_mtx mid-execution and re-acquires it at the end while leaving ch->running set and state ATA_ACTIVE - a late ata_interrupt/ata_reinit/ata_fail_requests holding state_mtx then blocks forever in the synchronous callout_cancel (ssleep 'costp') while the callback waits for the lock. If the callback wins instead, the same window double-completes the request. The comment claims ch->running is NULLed; the code never does it. REPRODUCED on guest with a KLD staging the exact interleaving (progress stuck at 3, canceller parked on 'costp', shutdown dies at 'syncing disks... 0'). Unpriv IO on a channel stalled >= its 10s timeout = permanent kernel wedge. NULL-ch->running-under-lock fix.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4606 Β· 10 files| File | Type | Description | Size | |
|---|---|---|---|---|
| cancelrace.c | β | 5.1 KB | view raw | |
| build.sh | β | 77 B | view raw | |
| run.sh | β | 181 B | view raw | |
| build.log | β | 1.0 KB | view raw | |
| run.log | β | 2.2 KB | view raw | |
| wedge-serial.log | β | 13.4 KB | view raw | |
| env.txt | β | 504 B | view raw | |
| VERDICT.md | β | 4.9 KB | β raw | |
| fix.diff | β | 1.8 KB | view raw | |
| verdict.json | β | 5.0 KB | view raw |
DF-4606 VERDICT
Status: reproduced (mechanism-level, deterministic) β impact: dos
What was run
Three guest runs (2026-09-22, DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 06:02:54 UTC 2026, X86_64_GENERIC, 6 vCPU, PIIX3 IDE w/ acd0, KVM):
- Run 1 (first module version): my canceller forgot to actually
lockmgr(&lk, LK_EXCLUSIVE)beforecallout_cancelβ the kernel panicked atlockmgr_cancel_begkern_lock.c:989KKASSERT((count & (LKC_XMASK|LKC_SMASK)) != 0)("lock must be held when issuing LK_CANCEL_BEG"). Accident, but useful: it provescallout_cancelon a mid-execution (INPROG) AUTOLOCK callout takes theLK_CANCEL_BEGcancel path (kern_timeout.c:892) while the callback is between its internalLK_RELEASEand re-acquire. The real nata callers (ata-lowlevel.c:442 from ata_interrupt, ata-all.c:208 from ata_reinit, ata-queue.c:578 from ata_fail_requests) all holdch->state_mtx, so they satisfy that assertion β and then deadlock as described below. - Run 2: harness had a softclock-thread
tsleep(..., timeout)β illegal in a softclock callback (its own timeout is serviced by itself). Replaced with a busy rendezvous. Even so the system wedged: userspacesleep 2never returned (RUN_RC=124), and orderly shutdown hung ("timeout expired. Shutdown terminated.", stuck at "syncing disks... 0" on the serial console β full excerpt in run.log). -
Run 3 (decisive): deterministic rendezvous +
hw.df4606_progresssysctl probe. Output (run.log):df4606: armed; watch for the wedge df4606: [softclock] callback: framework holds lk, dropping it (== ata-queue.c:557) df4606: [canceller] acquired lk inside the window (== late ata_interrupt with state_mtx) df4606: [canceller] calling callout_cancel() (== ata-lowlevel.c:442) ... hw.df4606_progress: 3 β callback entered the re-acquire lockmgr B5 costp df4606c β canceller parked in _callout_cancel_or_stop's ssleep kldunload: can't unload file: Device busy
progress=3 (set immediately before lockmgr(&lk, LK_EXCLUSIVE)) and
never 4 proves the softclock thread is permanently blocked in the
re-acquire β the exact statement of the finding. The canceller's wchan
costp is the literal sleep channel of kern_timeout.c:920. Orderly
shutdown of the wedged guest again failed (vm.sh down force-killed QEMU;
wedge-serial.log).
Why this is the finding's mechanism and not an artifact
- The KLD uses the same primitives the nata path uses:
callout_init_lk(&co, &lk)== ata-queue.c:93; the callback'sLK_RELEASE/LK_EXCLUSIVEpair == ata_timeout's ata-queue.c:557/560; the canceller's hold-the-lock +callout_cancel== ata-lowlevel.c:442 / ata-all.c:208 / ata-queue.c:578. - The kernel side is unmodified (stock guest kernel; the module only uses the public callout/lockmgr API exactly as ata does).
- No timing luck is needed in the KLD because the rendezvous pins the interleaving; the ATA code admits the same interleaving whenever the completion lands in its (unpinned, sub-ms) window after a β₯10 s stall.
Why not reproduced in-situ on the ATA path
The ATA trigger needs (a) a request that times out β the emulated PIIX3 channel completes 512 MB in 0.69 s (see DF-4607/stress.log), so timeouts never fire β and (b) the completion to land inside the callback's window. There is no user- or root-controllable way to stall QEMU's IDE from inside the guest (no CD medium, no monitor access from the guest). The window and all three interleaving entry paths are proven by the source trace in README.md; the interleaving semantics are proven by the KLD. In-situ exploitation is a hardware-reliability event (failing/slow disk, aggressive PM), after which the kernel wedges permanently.
Exploit chain
Not a corruption primitive; the deliverable is denial of service: any unprivileged IO can be the request that times out on a slow/failing channel; the kernel then deadlocks ch->state_mtx + one softclock thread, hanging the boot-disk channel and all timers that land on the wedged CPU's callwheel. No uid=0 route exists for this class.
Fix
fix.diff: ch->running = NULL; in ata_timeout while still holding
state_mtx (what the comment at ata-queue.c:551-553 already claims the code
does). fix_status=not_testable: validating it requires staging the in-situ
timeout+late-completion interleave, which the guest hardware cannot
produce; the KLD reproduces the primitive, not the ata path, so patching
ata-queue.c cannot change the KLD's outcome by construction.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testablefix.diff authored (ch->running = NULL under state_mtx). Not validatable on this guest: baseline requires the in-situ timeout+boundary-completion interleave the emulated hardware cannot stage, and the KLD reproduces the generic primitive, which the ata-queue.c patch cannot affect by construction.
findings/poc/DF-4606/fix.diff
Confirmed kernel references
- sys/dev/disk/nata/ata-queue.c:557
- sys/dev/disk/nata/ata-queue.c:560
- sys/dev/disk/nata/ata-queue.c:578
- sys/dev/disk/nata/ata-queue.c:93
- sys/dev/disk/nata/ata-lowlevel.c:442
- sys/dev/disk/nata/ata-all.c:208
- sys/dev/disk/nata/ata-all.c:333
- sys/kern/kern_timeout.c:609
- sys/kern/kern_timeout.c:888
- sys/kern/kern_timeout.c:920
- sys/kern/kern_lock.c:160
Detail
Exploit chain
unprivileged IO -> request armed with callout_init_lk(&request->callout, &ch->state_mtx) -> device stalls >= request->timeout (10 s) -> callout fires ata_timeout which releases state_mtx mid-callback -> late completion interrupt takes state_mtx and calls callout_cancel in ata_end_transaction -> canceller sleeps on costp holding state_mtx; callback blocks re-acquiring state_mtx -> permanent kernel deadlock: boot-disk channel dead + one softclock thread (and its CPU's callwheel) lost -> system wedge (timers and orderly shutdown hang).
Evidence (decisive lines)
findings/poc/DF-4606/run.log (progress=3 probe, wchan costp, kldunload EBUSY, RUN_RC=124 sleep, stuck 'syncing disks... 0'); wedge-serial.log (full serial console of run 3 incl. failed shutdown); VERDICT.md (run-by-run narrative incl. run-1 lockmgr_cancel_beg KKASSERT proving the INPROG cancel path); cancelrace.c (deterministic staging of the cited interleaving using the same callout_init_lk/LK_RELEASE/LK_EXCLUSIVE/callout_cancel API sequence)
PoC changes
Seed sketch did not exist (new finding); authored from scratch. Three harness iterations were needed: (1) canceller originally forgot lockmgr(LK_EXCLUSIVE) before callout_cancel -> exposed the LK_CANCEL_BEG 'lock must be held' KKASSERT instead; (2) callback slept with a timeout inside softclock (illegal: serviced by itself) -> replaced with busy rendezvous; (3) added hw.df4606_progress sysctl probe because kprintf from the post-wedge softclock context does not drain.
Verified recommended fix
In ata_timeout(), set ch->running = NULL while still holding ch->state_mtx (before LK_RELEASE) -- the comment at ata-queue.c:551-553 already claims this; it closes both the callout_cancel deadlock and the duplicate-completion variant.
Verdict
ata_timeout() (ata-queue.c:539-562) is a lock-aided callout callback that drops ch->state_mtx mid-execution and re-acquires it at the end, while leaving ch->running set and ch->state == ATA_ACTIVE. Any late ata_interrupt() reaching ata_end_transaction's callout_cancel (ata-lowlevel.c:442), or ata_reinit() (ata-all.c:208), or ata_fail_requests() (ata-queue.c:578), all holding state_mtx, then blocks forever: _callout_cancel_or_stop(sync=1) issues LK_CANCEL_BEG and ssleep("costp")s waiting for the callback to return while the callback is blocked re-acquiring the very lock the canceller holds (LKC_CANCEL does not abort a non-CANCELABLE acquisition, kern_lock.c). Deterministic KLD staging of the exact interleaving on the stock guest kernel reproduced the permanent circular wait (sysctl progress probe shows the softclock callback entered the re-acquire lockmgr and never returned; canceller parked on wchan costp; kldunload EBUSY), the loss of the wedged CPU's callwheel (userspace sleep RUN_RC=124), and the death of orderly shutdown (stuck at 'syncing disks... 0', force-kill required, twice). In-situ ATA trigger additionally needs a >=10 s channel stall then boundary completion, which the emulated PIIX3 cannot produce from inside the guest; window and entry paths proven by source trace. If the callback instead wins the re-acquire race, the same window yields a duplicate completion (second ATA_LF_UNLOCK + second ata_finish). Local DoS from unprivileged IO; no uid=0 route for this class.
No comments yet.