DF-2768 — unix98 pty clone-limit off-by-one: ptis[MAXPTYS] OOB read+write

What it is

ptyclone() (sys/kern/tty_pty.c:176) asks for a new unit with

unit = devfs_clone_bitmap_get(&DEVFS_CLONE_BITMAP(pty), MAXPTYS);

but devfs_clone_bitmap_get() only refuses units strictly greater than its limit (sys/vfs/devfs/devfs_helper.c:224-227):

unit = devfs_clone_bitmap_fff(bitmap);
if (limit > 0 && unit > limit)      /* unit == limit passes! */
        unit = -1;

With all units 0..999 allocated, fff returns exactly 1000, the check 1000 > 1000 is false, and unit 1000 is accepted. ptis is allocated with exactly MAXPTYS entries (tty_pty.c:1292: kmalloc(sizeof(struct pt_ioctl *) * MAXPTYS, ...) = 8000 bytes), so ptyclone then executes, with unit == 1000:

• tty_pty.c:186 if ((pti = ptis[unit]) == NULL) — out-of-bounds read of uninitialized heap memory, consumed as a pt_ioctl * pointer (type-confusion primitive if non-NULL);
• tty_pty.c:190 ptis[unit] = pti — out-of-bounds write of a kernel heap pointer 8 bytes past the requested allocation size.

Today the OOB read yields NULL by allocator accident: kmalloc(8000) rounds to an 8192-byte zone chunk and the M_ZERO path bzeros the full rounded chunk size (kern_slaballoc.c: zoneindex() rewrites the request to the chunk size; bzero(chunk, size) at the done: label), so the slack at +8000..8191 is zero. The OOB write therefore lands in dead slack of ptis' own chunk — no cross-object corruption at MAXPTYS=1000. If MAXPTYS were a power-of-two multiple (e.g. 1024 → exactly 8192 bytes), ptis[MAXPTYS] would instead read/write the first 8 bytes of the neighboring heap chunk — a groomable type-confusion/corruption primitive.

Observable effect on the stock kernel: a functional 1001st pty is created beyond the enforced limit; the OOB slot persists and is reused across cycles (ptis is never freed and pti_done() never clears ptis[unit]).

Reproduce (unprivileged)

cc -O2 -Wall -o ptmx_limit_offbyone ptmx_limit_offbyone.c
./ptmx_limit_offbyone          # as ANY user (e.g. nobody)
# expected (vulnerable): "VULNERABLE: /dev/pts/1000 exists ...", rc=2

Also in the pack:

• probe_oob_unit.c — holds 1000 ptys, shows the 1001st clone creating /dev/pts/1000, the master open failing (ENODEV via the autoclone fallback), /dev/pts/1000 opening successfully as a slave, and the 1002nd open being refused.
• cycle_oob_unit.c — cycles the OOB unit 5×: each cycle re-creates /dev/pts/1000 and reuses the same pti (vmstat -m "ptys" count stays at 1002 = ptis array + 1001 ever-allocated ptis), proving the OOB slot write at ptis[1000] persisted.

Success criterion: /dev/pts/1000 exists while 1000 ptys are held → ptis[1000] OOB taken. Fixed kernel: open #1001 refused (ENODEV), no /dev/pts/1000, exactly ≤1000 concurrent ptys.

Fix

MAXPTYS - 1 is the highest valid unit. See fix.diff.

DF-2768 VERDICT

Status: reproduced (unprivileged, default config, stock INVARIANTS kernel #0). Impact: security-limit bypass + latent out-of-bounds read/write (memcorrupt class); no crash/corruption observable at MAXPTYS=1000 because the OOB lands in allocator slack.

How it was proven

1. Static: devfs_clone_bitmap_get() (devfs_helper.c:221-234) accepts unit == limit; ptyclone passes the array size MAXPTYS as the limit; ptis[unit] at tty_pty.c:186/190 indexes past the 1000-entry array (8000 bytes, tty_pty.c:1292). fff returns 1000 when units 0..999 are busy (lowest-free-bit scan, bitmap chunk 15 bits 40..63 free).
2. Dynamic (as nobody): ptmx_limit_offbyone opened 1000 ptmx fds; the 1001st open failed at the devfs layer (ENODEV, autoclone fallback default_dev_ops→noopen) but the clone had already executed: /dev/pts/1000 existed (stat proof), i.e. unit 1000 was allocated through the OOB slot.
3. The OOB pty is real: open("/dev/pts/1000", O_RDWR|O_NONBLOCK) succeeded (probe_oob_unit.c) and was close-cycled repeatedly (cycle_oob_unit.c ×5) — a working slave beyond MAXPTYS.
4. The OOB write persisted: vmstat -m M_PTY ("ptys") count stayed 1002 (ptis array + 1001 ever-allocated ptis) across 5 create/destroy cycles of unit 1000 — the pti stored at ptis[1000] is reused, so the first cycle's ptis[1000] = pti write survived.
5. Why it doesn't crash today: the first OOB read returned NULL (M_PTY count rose by exactly one pti ever), because kmalloc's M_ZERO bzero covers the full rounded 8192-byte chunk (zoneindex() rounds 8000→8192 and _kmalloc bzeros the rounded size — kern_slaballoc.c zoneindex + done: bzero). A non-NULL uninit value would have been used directly as struct pt_ioctl * by tty_pty.c:202-218 (pti->devc = make_only_dev(...) etc. — wild writes). With MAXPTYS=1024 the slot would fall into the next chunk (cross-object corruption, groomable neighbor).

Exploit chain

Not pursued beyond proof of the OOB: on the current constant (1000) the read is deterministically NULL and the write lands in intra-chunk slack, so the present-day impact is the limit bypass (a 1001st pty exists and is slave-usable) plus a latent CWE-125/457/787 that becomes cross-object the moment MAXPTYS is changed to a zone-boundary multiple or the allocator behavior changes. Classified dos-level impact.

Fix validation

Combined patch (DF-2768 + DF-2769 hunks) built with make -j6 nativekernel && make installkernel in the guest; both PoCs re-run: ptmx_limit_offbyone then reported open #1001 refused and /dev/pts/1000 absent (see fix run logs). fix.diff = the MAXPTYS-1 hunk alone.

Kernel references

• sys/kern/tty_pty.c:176 (limit passed = array size)
• sys/kern/tty_pty.c:186,189-190 (ptis[unit] OOB read/write)
• sys/kern/tty_pty.c:127 (MAXPTYS definition)
• sys/kern/tty_pty.c:1292-1293 (ptis = kmalloc(MAXPTYS entries))
• sys/vfs/devfs/devfs_helper.c:221-234 (unit > limit off-by-one)
• sys/kern/kern_slaballoc.c zoneindex() + _kmalloc done: (why the OOB read is zeroed today: chunk-rounding + full-chunk M_ZERO bzero)