p_mqueue_cnt accounting broken across fork(): decrement runs in the last-reference closer (not the opener) while fork zeroes the child's counter — u_int underflow defeats the '== mq_open_max' check, and the open-existing path has no check at all
| Field | Value |
|---|---|
| ID | DF-2780 |
| Status | new |
| Severity | Medium |
| CVSS 3.1 | CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:L |
| CWE | CWE-191 Integer Underflow (CWE-770) |
| File | sys/kern/sys_mqueue.c |
| Lines | 384 (decrement), 452/577 (equality checks), 531-563 (unchecked found path), 592 (increment) |
| Area | kern |
| Confidence | certain |
| Discovered | 2026-08-31 |
| Pass | 2 (GLM 5.3 second pass) |
| Bucket | base:kern |
| Reported | pending |
| Known CVE | none |
| CVE match | novel |
Summary
p_mqueue_cnt (u_int) is incremented in the opener but decremented in
mq_close_fop() in whichever process drops the LAST file reference —
with fork() the child starts at zero yet inherits descriptors. A child
closing one inherited descriptor as last reference underflows
0 → 0xffffffff, after which the equality check == mq_open_max never
matches: the child created 513 queues past the 512 limit
(demonstrated). Independently, the open-EXISTING path increments the
counter with NO limit check — mq_open(existing) succeeds at the EMFILE
limit (demonstrated). The opener-side mirror bug leaks its counter
(premature EMFILE). Enables unbounded queue creation in combination
with DF-2781.
Proof of contest
VERIFIED on the stock guest (findings/poc/DF-2780/fork_cnt.c): control 512/EMFILE; "[existing] mq_open at/past EMFILE limit: SUCCEEDED"; "[fork-child] created 513 queues, exit=42 => BYPASS REPRODUCED". Fix (inherit the count across fork by counting DTYPE_MQUEUE fds after fdcopy; use >=; check the existing path) in the pack.
Recommended fix
See findings/poc/DF-2780/fix.diff (kern_fork.c count-inherit hunk +
= comparisons + found-path check).
Timeline
- 2026-08-31 Discovered during pass-2 audit of sys_mqueue.c (GLM 5.3); bypass reproduced unpriv same run.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-2780 · 10 files| File | Type | Description | Size | |
|---|---|---|---|---|
| fork_cnt.c | — | 4.8 KB | view raw | |
| build.sh | — | 40 B | view raw | |
| run.sh | — | 174 B | view raw | |
| build.log | — | 151 B | view raw | |
| run.log | — | 490 B | view raw | |
| fix_validation.log | — | 643 B | view raw | |
| VERDICT.md | — | 1.5 KB | ↓ raw | |
| verdict.json | — | 3.0 KB | view raw | |
| fix.diff | — | 1.7 KB | view raw | |
| env.txt | — | 119 B | view raw |
VERDICT — DF-2780: p_mqueue_cnt accounting broken across fork → EMFILE bypass
Status: REPRODUCED · impact: dos (per-process mqueue limit bypass enabling unbounded queue creation) · confidence: certain
What was proven on the guest (stock kernel, build #0)
./fork_cnt output (run.log):
- control: fresh process creates exactly 512 queues → EMFILE
(kern.mqueue.mq_open_max=512);
- open-existing path: mq_open(existing) at/past EMFILE limit: SUCCEEDED
— no limit check exists on this path at all (sys_mqueue.c:531-563);
- fork path: child closes ONE inherited descriptor as the last file
reference (pipe-synchronized after the parent closed its copy) — its
u_int p_mqueue_cnt (proc.h:235) underflows 0 → 0xffffffff in
mq_close_fop() (sys_mqueue.c:384) — then creates 513 queues before
hitting EMFILE (the equality check == mq_open_max at sys_mqueue.c:452
misses the wrapped counter; 0xffffffff + k wraps past 512). Exit code 42:
BYPASS REPRODUCED.
Fix validation (patched kernel, build #1 04:57)
fix.diff: (1) kern_fork.c inherits the mqueue count across fork (counts
DTYPE_MQUEUE descriptors after fdcopy); (2) >= instead of == at both
checks; (3) limit enforced on the open-existing path.
Patched output (fix_validation.log):
- control: unchanged (512/EMFILE);
- existing path: Too many open files (was: SUCCEEDED);
- fork path: child creates 512 (was 513), bypass not observed,
exit=1 (was 42).
fix_status: fixed (fix_baseline_reproduced=1, fix_patched_reproduced=0).
Fix verification
fixedPatched kernel: existing-path open now returns EMFILE at the limit; fork child creates exactly 512 (no underflow effect) and exits 1 ('bypass not observed').
fix_validation.log
Confirmed kernel references
Detail
Exploit chain
open 1 queue -> fork -> parent closes its copy (fo_close NOT run; parent count leaks) -> pipe-sync -> child closes inherited copy as LAST ref -> mq_close_fop decrements child's 0 counter to 0xffffffff -> equality limit check misses -> unlimited creates until fd limit; independently, opening existing queues bypasses the check without any fork.
Evidence (decisive lines)
["run.log: control 512/EMFILE, '[existing] ... SUCCEEDED', '[fork-child] created 513 queues ... exit=42 => BYPASS REPRODUCED'", 'fix_validation.log: patched - existing path EMFILE, fork-child exactly 512, exit=1']
PoC changes
Pipe-synchronized close ordering so the child's close is the last reference (first version closed in the wrong order and did not underflow); control-loop fds closed/unlinked between stages; raw syscalls.
Verified recommended fix
Inherit p_mqueue_cnt across fork (count DTYPE_MQUEUE fds after fdcopy), use >= instead of ==, and enforce the limit on the open-existing path (fix.diff).
Verdict
Per-process mqueue limit bypass proven three ways on the stock kernel: (1) control = EMFILE at exactly 512 creates; (2) the open-EXISTING path performs no limit check at all - mq_open(existing) succeeded at the EMFILE limit; (3) fork path - a child that drops the last file reference of one inherited descriptor underflows its u_int p_mqueue_cnt 0->0xffffffff (decrement runs in the last-ref closer, not the opener; fork zeroes the child counter) after which the equality check '== mq_open_max' never matches: the child created 513 queues (exit 42 = bypass). Combined with DF-2781 this enables unbounded queue/memory creation; combined with repeated fork+close rounds the per-process limit is void.
No comments yet.