Summary

radix_max() computes (1UL << (root->height * RADIX_TREE_MAP_SHIFT)) - 1. On 64-bit platforms RADIX_TREE_MAX_HEIGHT = howmany(64, 6) = 11, so when height == 11 the expression is 1UL << 66 — a shift by more than the width of unsigned long, which is undefined behavior per ISO C11 §6.5.7. The expand loop in radix_tree_insert can drive height to exactly 11 (it checks == RADIX_TREE_MAX_HEIGHT only before incrementing at line 196, allowing the increment to 11 at line 211), after which every subsequent radix_max() call on that root is UB. On x86-64 the hardware masks the shift count to 6 bits (66 & 63 = 2), yielding radix_max == 3, which causes the loop to re-enter and hit the E2BIG check — benign but non-portable. No memory corruption occurs, but the tree is left in an inconsistent state (height=11) after a failed insert of an impossibly large index.

Root cause

• radix-tree.h:39-40 defines RADIX_TREE_MAX_HEIGHT as howmany(sizeof(long)*NBBY, RADIX_TREE_MAP_SHIFT) = howmany(64, 6) = 11.
• linux_radix.c:45 computes 1UL << (root->height * 6). At height == 11 this is 1UL << 66.
• The guard in radix_tree_insert at linux_radix.c:196 (if (root->height == RADIX_TREE_MAX_HEIGHT) return -E2BIG) fires only after height has already been incremented to 11 at line 211 in the prior iteration — meaning the while-condition at line 193 (radix_max(root) < index) is re-evaluated with height == 11, invoking the UB.
• The same UB then affects any subsequent lookup/insert/delete on this root via their radix_max() calls at lines 64, 88, 126, 193.

Threat model & preconditions

• Attacker position: N/A — not exploitable through any reachable DRM attack surface.
• Privileges gained or impact: none. ISO C undefined behavior (correctness/portability risk on non-x86 architectures where large shifts may trap or yield 0); a leaked/inconsistent tree node when E2BIG is returned after the expand loop already allocated a new root level (minor memory leak).
• Required config or capabilities: the height-11 state requires inserting an unsigned long index > (1UL<<60)-1 (~1.15e18). All DRM callers use bounded indices: i915_gem_execbuffer.c:776 passes u32 handle (max 2^32, needing height 6); i915_gem.c:6569 iter->radix uses page-offset indices bounded by GEM object size (max ~2^20). A malicious kernel module (already root-privileged) could theoretically trigger it, but that is not a privilege-escalation path.
• Reachability: kernel-internal API misuse only.

Proof of Concept

Not practically exploitable for memory corruption or privilege escalation. To demonstrate the UB on x86-64 (informational only, not a security PoC): write a kernel module that calls radix_tree_insert(root, 1UL<<63, &dummy) on a fresh tree — the expand loop drives height 1→…→11, the 1UL<<66 UB manifests as radix_max returning 3 (masked shift), and -E2BIG is returned leaving root->height == 11 in an inconsistent state. No panic, no corruption. Reproducing on a non-x86 architecture (e.g., older ARM where large shifts may yield 0) could cause the expand loop to fail to terminate until the height check trips, still resulting only in -E2BIG.

Impact

No memory corruption, no info leak, no privilege escalation. Portability / correctness hardening defect only.

Recommended fix

Make radix_max() robust against shift overflow by clamping the shift to the word width. When height*RADIX_TREE_MAP_SHIFT >= BITS_PER_LONG, return ULONG_MAX (all indices representable), which is the mathematically correct maximum for a tree tall enough to index the full unsigned long range. This also makes the tree correctly accept all 64-bit indices without ever hitting the E2BIG path for valid pointer-keyed indices.

--- a/sys/dev/drm/linux_radix.c
+++ b/sys/dev/drm/linux_radix.c
@@ -42,7 +42,11 @@
 static inline unsigned long
 radix_max(struct radix_tree_root *root)
 {
-   return ((1UL << (root->height * RADIX_TREE_MAP_SHIFT)) - 1UL);
+   unsigned int shift = root->height * RADIX_TREE_MAP_SHIFT;
+
+   if (shift >= sizeof(unsigned long) * NBBY)
+       return (~0UL);
+   return ((1UL << shift) - 1UL);
 }

 static inline int

References

• ISO C11 §6.5.7 — "If the value of the right operand is negative or is greater than or equal to the width of the promoted left operand, the behavior is undefined."
• sys/dev/drm/radix-tree.h:39-40 — RADIX_TREE_MAX_HEIGHT = 11 on 64-bit.

Timeline

• 2026-07-25 Discovered during automated audit.
• 2026-07-25 Reported to DragonFlyBSD security contact.