DF-4414 — radeon_vm_fini() DragonFly livelock: while (vm->va.rb_node) spins forever when radeon_bo_reserve() fails

Build

cc -O2 -Wall -o /tmp/trigger_fini DF-4414/trigger_fini.c
# radeon_abi.h must sit next to the .c

Run

/tmp/trigger_fini

Requires a cayman-or-newer radeon KMS device and two cooperating processes sharing a BO (GEM flink).

Expected result on capable hardware

• Parent: maps a flink-shared BO into its vm, keeps submitting CS so the BO's TTM reservation is (repeatedly) held via ttm_eu_reserve_buffers.
• Children: GEM_OPEN the shared BO, map it into their own vm, then exit with SIGKILL racing the close. A child's postclose → radeon_vm_fini must reserve the shared BO; the reservation is held by the parent and the child has a pending signal → ttm_bo_reserve(..., interruptible=1) returns -ERESTARTSYS immediately, the DFly while (vm->va.rb_node) loop makes no progress and never exits → the child's thread spins forever in kernel mode (unkillable), the drm fd never closes, and the vm is leaked. Parent observes waitpid() never returning.

Observed on THIS audit guest

No radeon GPU (/dev/dri absent). Trigger builds and runs, exits at device open (run.log). Mechanism proven by trace in VERDICT.md — the divergence from upstream is DFly-specific.

DF-4414 — VERDICT

status: not_reproduced (hardware absent) — DFly-specific divergence from upstream; mechanism certain by trace

Root cause (certain)

DFly replaced upstream's rbtree_postorder_for_each_entry_safe() walk in radeon_vm_fini() with a retry-forever loop over the interval tree root (introduced when the DFly interval-tree mock — a circular list, see sys/dev/drm/linux_interval_tree.c — made the postorder RB iterator unusable):

1250:   while (vm->va.rb_node) {
1251:       bo_va = container_of((void *)vm->va.rb_node, struct radeon_bo_va, it);
1253:       r = radeon_bo_reserve(bo_va->bo, false);
1254:       if (!r) {
1255:           interval_tree_remove(&bo_va->it, &vm->va);
...         kfree(bo_va);
1260:       }
1261:   }

If radeon_bo_reserve() fails, the tree root is unchanged → the loop repeats the identical failing operation forever.

radeon_bo_reserve(bo, false) is ttm_bo_reserve(&bo->tbo, /*interruptible=*/!false, /*no_wait=*/false, NULL) (radeon_object.h:64-68): when the BO's reservation object is held by another thread and the current thread has a pending signal it returns -ERESTARTSYS immediately, without blocking. The closing thread is exiting precisely because it was signalled → tight kernel-mode spin loop, no blocking, no exit condition:

• the exiting thread never returns to user mode → unkillable,
• the drm file is never released, the radeon_vm never freed (leak),
• one CPU pegged at 100% in kernel space.

Upstream Linux performs a single pass (rbtree_postorder_for_each_entry_safe) and merely leaks the bo_va on reserve failure (dev_err "still active bo inside vm" at radeon_vm.c:1237 mirrors that). DFly's rewrite turned a bounded leak into an unbounded livelock. This is a genuine DFly-local defect, not an inherited one.

Reachability

Requires vm->va to be non-empty at radeon_vm_fini() and a reserve failure:

• Normal exits empty the vm first (GEM object teardown calls radeon_vm_bo_rmv), BUT a BO shared with another process via GEM flink / dma-buf keeps the radeon_bo alive, so this fd's bo_va stays in the tree at close time (radeon_gem.c free path only rmv's when the object dies).
• The other process keeps the shared BO's reservation held across CS parsing (ttm_eu_reserve_buffers → radeon_bo_list_validate, radeon_cs.c:184-190) while the victim exits under a signal.

All steps are ordinary unprivileged DRM operations.

Why not reproduced here

No radeon GPU on the audit guest (std VGA only). Trigger builds and runs, exits at open("/dev/dri/card0") (run.log). Severity Medium (local DoS: permanently wedged kernel thread + unkillable fd + vm leak; repeatable).

Recommended fix

Make progress mandatory: use an uninterruptible reserve for teardown and bail out (leak, like upstream) rather than spinning if it still fails. See fix.diff.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.