drm_sched_fini returns without cancelling the armed work_tdr or draining ring_mirror_list β delayed-work/callout executes on the freed scheduler after driver teardown (softclock panic)
Summary
fini only stops the thread; with any job pending, work_tdr is armed and fini neither cancels it nor drains the list - callers free the scheduler container immediately, leaving the callout to fire on freed memory. REPRODUCED on guest: fini + kfree exactly as amdgpu frees amdgpu_ring -> ~2s later panic in softclock_handler. Privileged trigger (unload/rebind/reset-failure teardown); freed-memory execution unconditional once fini runs with a pending TDR. cancel_delayed_work_sync fix validated on rebuilt amdgpu.ko.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4741 Β· 11 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 1.2 KB | β raw | |
| VERDICT.md | β | 2.7 KB | β raw | |
| build.sh | β | 103 B | view raw | |
| run.sh | β | 135 B | view raw | |
| build.log | β | 75 B | view raw | |
| run_fixed.log | β | 2.1 KB | view raw | |
| panic.txt | β | 2.7 KB | view raw | |
| env.txt | β | 290 B | view raw | |
| fix.diff | β | 1.6 KB | view raw | |
| verdict.json | β | 3.6 KB | view raw | |
| ../DF-4740/df4740.c | β | 14.9 KB | view raw |
DF-4741 β drm_sched_fini returns with an armed work_tdr / undrained job list β post-teardown UAF
Build
Same harness as DF-4740 (df4740.c, Makefile) β see ../DF-4740/README.md.
Run
kldload amdgpu kldload /tmp/df4740/df4740.ko sysctl hw.df4740_test=2
Expected (baseline, unpatched scheduler β observed, panic.txt)
[DF4741] job ... in flight, work_tdr armed (timeout=200 ticks); calling drm_sched_fini()
[DF4741] drm_sched_fini() returned WITHOUT cancelling work_tdr ...
[DF4741] reclaim block 0 at 0xfffff8008f844440 (zeroed) <- exact freed sched address
...
panic: _callout 0xfffff8008f968b80 verifier 0xfffff8008f844630 toc 0 (expected ...) func ...
softclock_handler() at softclock_handler+0x502
Debugger("panic")
The armed work_tdr delayed-work callout fires on the freed (and reclaimed)
scheduler ~1-2 s after drm_sched_fini() returned and the caller freed the
scheduler, exactly as amdgpu_fence_driver_fini() (amdgpu_fence.c:495-519)
frees struct amdgpu_ring (which embeds ring->sched) right after
drm_sched_fini(&ring->sched).
Post-fix (fix.diff adds cancel_delayed_work_sync to drm_sched_fini)
run_fixed.log: same test on the patched amdgpu.ko β no callout fire, no
panic, guest stays up ("STILL-UP-4s").
DF-4741 VERDICT β drm_sched_fini leaves armed TDR / pending jobs on teardown
Status: REPRODUCED (panic via softclock on freed scheduler; fix validated)
Root cause
drm_sched_fini() β sys/dev/drm/scheduler/sched_main.c:652-656:
void drm_sched_fini(struct drm_gpu_scheduler *sched)
{
if (sched->thread)
kthread_stop(sched->thread);
}
It neither cancels sched->work_tdr (armed by drm_sched_start_timeout
whenever jobs sit on ring_mirror_list, sched_main.c:194-199) nor drains the
job list, and takes no reference-participation in pending finish_work
items. Every in-tree caller frees the scheduler's container right after:
amdgpu_fence_driver_fini() calls drm_sched_fini(&ring->sched)
(amdgpu_fence.c:509) and the enclosing struct amdgpu_ring is released with
the device during teardown. The pending delayed work / callout then executes
against freed memory.
Reproduction (guest, harness test 2 β df4740.c)
From a fresh reset (panic.txt):
- Scheduler initialized with a 2 s timeout; a hung job sits on
ring_mirror_list, sojob_beginarmedwork_tdr(sched_main.c:246). drm_sched_fini()called (thread stopped; work_tdr STILL armed).- Scheduler
kfree()d and the block reclaimed with zeroed allocations (reclaim block 0 landed exactly on the freed scheduler address) β the stand-in for amdgpu freeingstruct amdgpu_ring. - ~2 s later the callout fired on the freed object:
panic: _callout ... verifier ... (expected ...) ... softclock_handlerβ guest down at db>.
Even when the callout verifier does not trip first, the queued
drm_sched_job_timedout work item itself runs on freed memory (list walk,
lockmgr(&sched->job_list_lock) inside the freed block, ops->timedout_job
indirect call) β the same class with RIP-control-grade consequences on this
guest.
Severity context: teardown is reached via driver unload/rebind (privileged) or device removal/reset-failure paths, hence Medium rather than High; the panic and the freed-memory execution are unconditional once fini runs with a pending TDR.
Fix
fix.diff hunk 3 (shared with DF-4740): cancel_delayed_work_sync(&sched->work_tdr)
after kthread_stop in drm_sched_fini. Validated: patched amdgpu.ko, test 2
re-run β no callout fire, no panic, guest stays up (run_fixed.log). A full
upstream-grade fix should additionally drain ring_mirror_list (signal +
free leftover jobs) as Linux 5.x does.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedfixed: patched amdgpu.ko rebuild; test 2 re-run shows no callout fire on the freed scheduler, no panic, guest stays up (run_fixed.log) vs baseline softclock verifier panic (panic.txt).
fix.diff; run_fixed.log; ../DF-4740/fix_build.log
Confirmed kernel references
Detail
Exploit chain
driver teardown (module unload / rebind / GPU removal) with any job pending on ring_mirror_list -> drm_sched_fini returns without cancelling work_tdr -> caller frees scheduler container (amdgpu_ring) -> armed delayed work fires on freed+reclaimed memory -> drm_sched_job_timedout executes from freed struct: freed-memory lockmgr, ring_mirror_list walk on dangling head, ops->timedout_job call through the freed sched->ops pointer.
Evidence (decisive lines)
panic.txt (test2 sequence + 'panic: _callout ... verifier' + softclock_handler trace, db> wedge); run_fixed.log (patched module: no fire, STILL-UP); ../DF-4740/df4740.c (harness); VERDICT.md
PoC changes
Shares the DF-4740 harness; test-2 timing tuned after the first run showed the TDR firing before fini (2*hz timeout + 2-tick pre-fini wait) and the reclaim switched from 0xde fill to zeroed blocks because queue_work skips entries whose on_queue/running flags are truthy.
Verified recommended fix
cancel_delayed_work_sync(&sched->work_tdr) in drm_sched_fini after kthread_stop (fix.diff hunk 3; full upstream-grade fix should also drain ring_mirror_list).
Verdict
Reproduced on a fresh-reset guest: drm_sched_fini (sched_main.c:652-656) only calls kthread_stop β it does not cancel the armed work_tdr delayed work and does not drain ring_mirror_list. The harness (test 2 of df4740.c) left a hung job on the list (work_tdr armed at sched_main.c:246), called drm_sched_fini, then kfree()d the scheduler exactly as amdgpu_fence_driver_fini frees struct amdgpu_ring after drm_sched_fini(&ring->sched) (amdgpu_fence.c:495-519); the block was reclaimed by a zeroed allocation at the identical address. ~2s later the armed callout fired on the freed object and softclock's verifier tripped: 'panic: _callout ... verifier ... (expected ...) ... softclock_handler', guest down at db> (panic.txt). Absent the verifier trip, the queued drm_sched_job_timedout work itself executes on freed memory (lockmgr on freed job_list_lock, ops->timedout_job indirect call). Trigger is teardown-time (driver unload/rebind/device removal β privileged), hence Medium; the freed-memory execution is unconditional once fini runs with a pending TDR. Fix (cancel_delayed_work_sync after kthread_stop, fix.diff hunk 3) validated: patched amdgpu.ko, same test -> no callout fire, no panic, guest stays up (run_fixed.log).
No comments yet.