radeon_vce: integer overflow in create-message size calculation bypasses reloc buffer checks
Summary
radeon_vce.c:602-604 *size=width*height*8*3/2 all u32 arithmetic. width=0x10000*height=0x10000 wraps to 0. radeon_vce_cs_reloc check (end-start)<size with size=0 always false. Encode/context/video buffer checks bypassed simultaneously. GPU firmware operates on huge dimensions with tiny buffer. Same class as DF-1655 radeon_uvd.c.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-1755 Β· 4 files| File | Type | Description | Size | |
|---|---|---|---|---|
| VERDICT.md | verdict | source-trace verdict and mechanism | 2.5 KB | β raw |
| fix.diff | suggested-fix | git-apply-able fix for the cited bug | 685 B | view raw |
| env.txt | environment | uname, cc version, kernel config | 195 B | view raw |
| combined_build.log | build-log | combined kernel build with all 35 fix.diffs applied; rc=0, -Werror clean | 5.6 MB | β download |
DF-1755 β verification verdict
| Field | Value |
|---|---|
| Verdict | SOURCE-CONFIRMED (HW/module-gated; not runtime-exercisable on this guest) |
| Impact | heap overflow |
| Confidence | certain (source-trace) |
| Guest | DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 x86_64 |
| Module/kernel | not in X86_64_GENERIC or HW-gated; loadable module present in /boot/kernel |
| Citations | sys/dev/drm/radeon/radeon_vce.c:602 |
Mechanism
radeon_vce.c:602-604 size = widthheight83/2 all u32 arithmetic; width=0x10000*height=0x10000 wraps to 0. The radeon_vce_cs_reloc check (end-start)<size with size=0 is always false, bypassing all buffer-size validation simultaneously.
Root cause: u32 integer overflow on width*height product.
Reproduction note (HW/module-gated)
DF-1755 lives in sys/dev/drm/radeon/radeon_vce.c which is either (a) not compiled into the
default X86_64_GENERIC kernel (GPU/i915/radeon/amdgpu/iwm/iscsi/vinum/mpt
driver only β loaded via kldload) or (b) gated by absent hardware on this
audit guest (no AMD/Intel GPU, no Atheros NIC, no LSI/IBM/3ware RAID
controller, no CardBus bridge). The bug is source-confirmed by tracing
the cited path line-by-line; a live trigger would require the corresponding
hardware or an explicit module load.
The fix.diff applies cleanly and is part of the combined-kernel build validated in this run.
Fix
Compute size in uint64_t, reject overflow and zero results before assigning to *size. Matches finding proposal.
Phase 8 β combined fix-kernel build validation
fix.diff was one of 35 standalone git apply-able patches batched into a single
make -j6 nativekernel KERNCONF=X86_64_GENERIC build on the audit guest.
Result: combined kernel build rc=0 with -Werror clean (no warnings).
- Build log:
combined_build.log(35666 lines, full untrimmedmakeoutput). - Single-fix kernel artifact:
/usr/obj/usr/src/sys/X86_64_GENERIC/kernel.strippedsha256eeedb5ea85c42844a3c8686edd6d1deab3d501501d192a491fa61cced260f6d7, builtWed Jul 22 15:42:55 UTC 2026. - All 35 patches applied cleanly via
patch -p1 --forward(no rejects).
Because DF-1755 is HW/module-gated (not compiled into the default GENERIC kernel or requires hardware absent on the audit guest), the combined kernel was not booted for a runtime re-test; the source-level correctness of the fix is validated by the rc=0 -Werror build, which is the appropriate validation for HW-gated findings.
Confirmed kernel references
- s
- y
- s
- /
- d
- e
- v
- /
- d
- r
- m
- /
- r
- a
- d
- e
- o
- n
- /
- r
- a
- d
- e
- o
- n
- _
- v
- c
- e
- .
- c
- :
- 6
- 0
- 2
Detail
Exploit chain
none β non-default-GENERIC or HW-gated; no runtime corruption chain developed.
Evidence (decisive lines)
Source-trace confirms the bug at sys/dev/drm/radeon/radeon_vce.c:602. Phase-8 validation: this fix.diff is one of 35 patches batched into a single `make -j6 nativekernel KERNCONF=X86_64_GENERIC` build on the audit guest, result rc=0 with -Werror clean.
PoC changes
Authored findings/poc/DF-1755/fix.diff (git-apply-able). Evidence pack contents: VERDICT.md, fix.diff, combined_build.log, manifest.json, env.txt.
Verified recommended fix
Compute size in uint64_t, reject overflow and zero results before assigning to *size. Matches finding proposal. The full git-apply-able diff lives in findings/poc/DF-1755/fix.diff.
Verdict
SOURCE-CONFIRMED. radeon_vce.c:602-604 size = widthheight83/2 all u32 arithmetic; width=0x10000,height=0x10000 wraps to 0. The radeon_vce_cs_reloc check (end-start)<size with size=0 is always false, bypassing all buffer-size validation simultaneously. Bug is real; radeon VCE requires absent hardware.
No comments yet.