⬢ DragonFlyBSD Kernel Audit
← triage · dashboard
DF-2781

No system-wide or per-user POSIX mqueue accounting: queues and their queued messages persist after the last close with no enumeration — unprivileged, unbounded, permanent-until-reboot kernel heap exhaustion

Field Value
ID DF-2781
Status new
Severity Medium
CVSS 3.1 CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H
CWE CWE-770 Allocation of Resources Without Limits
File sys/kern/sys_mqueue.c
Lines 71-76 (only limits), 391-399 (destroy gated on MQ_UNLINK)
Area kern
Confidence certain
Discovered 2026-08-31
Pass 2 (GLM 5.3 second pass)
Bucket base:kern
Reported pending
Known CVE none
CVE match novel

Summary

mq_close_fop() destroys a queue only when MQ_UNLINK is set; queued messages are kmalloc'd (M_MQBUF) and survive the creator's exit. The only limits are per-process open count and per-queue geometry — nothing bounds the system-wide queue count or queued bytes, and there is no way to enumerate queues, so memory stranded under withheld names is unreclaimable short of reboot. Demonstrated: 96.4 MB stranded then retained after process exit; sustained stranding drove the kernel to ENOMEM on mq_send/mq_open at ~390 MB on a 4 GB guest with 2.48 GB user memory free, with zero reclaim progress. Linux ships RLIMIT_MSGQUEUE + fs.mqueue.* sysctls for exactly this attack class.

Proof of contest

VERIFIED on the stock guest (findings/poc/DF-2781/strand.c): ./strand brief → 96.4 MB persists after exit; ./strand exhaust → "mq_send #280: Cannot allocate memory" then mq_open ENOMEM at ~390 MB, permanent. Fix (kern.mqueue.mq_max_queues sysctl, default 256, maintained counter enforced at creation) validated with sysctl=8 → strander stops after exactly 8 queues.

See findings/poc/DF-2781/fix.diff; upstream should also consider per-uid byte limits (RLIMIT_MSGQUEUE model) and an enumeration/cleanup interface.

Timeline

  • 2026-08-31 Discovered during pass-2 audit of sys_mqueue.c (GLM 5.3); permanent exhaustion reproduced unpriv same run.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-2781 · 11 files
FileTypeDescriptionSize
strand.c — 3.0 KB view raw
build.sh — 36 B view raw
run.sh — 291 B view raw
build.log — 151 B view raw
run.log — 412 B view raw
run.exhaust.log — 5.1 KB view raw
fix_validation.log — 490 B view raw
VERDICT.md — 2.4 KB ↓ raw
verdict.json — 3.5 KB view raw
fix.diff — 1.6 KB view raw
env.txt — 119 B view raw
VERDICT.md
↓ download raw

VERDICT — DF-2781: no system-wide mqueue cap → persistent unpriv kmem exhaustion

Status: REPRODUCED · impact: dos · confidence: certain

What was proven on the guest (stock kernel, build #0, 4 GB RAM)

  1. Persistence: ./strand brief strands 96.4 MB (12 queues × 512 msgs × 16 KB); after the strander EXITED, vmstat -m shows mqueues ... 97.5M still allocated (run.log) — the memory belongs to orphaned queues; POSIX semantics keep them (and their messages) alive until mq_unlink (sys_mqueue.c:391-399), and there is no enumeration interface, so queues whose names are withheld can never be reclaimed short of reboot.
  2. Exhaustion: ./strand exhaust ran until the KERNEL began failing allocations: mq_send #280: Cannot allocate memory, then mq_open: Cannot allocate memory (M_WAITOK kmalloc returning NULL) at ~390 MB stranded (run.exhaust.log). 2.5 min of retry loops made zero progress (ceiling reached); vmstat still showed 2.48 GB of userland memory free — the failure is kernel-heap-side. Basic system ops still worked at that point (cc, file create), but the mqueue subsystem was permanently dead and 390 MB permanently consumed on a 4 GB machine.

Each queue may legally hold 512 × ~16.4 KB ≈ 8.4 MB; a single process may open 512 such queues (~4.3 GB), exit, and repeat with new names. Linux mitigates exactly this attack class with RLIMIT_MSGQUEUE + fs.mqueue.* sysctls; DragonFly has only per-process open count and per-queue geometry caps (sys_mqueue.c:71-76).

Fix validation (patched kernel, build #1 04:57)

fix.diff adds kern.mqueue.mq_max_queues (default 256, root-tunable), enforced at creation, with a maintained global counter (decremented at both LIST_REMOVE sites).

Validation with the sysctl set to 8 (fix_validation.log): the same strand exhaust stops after exactly 8 queues / 64.2 MB (mq_open: Cannot allocate memory at #9), vmstat confirms 64.0 M held, and a second strander cannot create even one queue. Sysctl restored to 256.

Note: a first fix iteration placed the cap check before the mq_mtx acquisition and goto exited with mq == NULL, tripping the shared exit path (lockmgr on a NULL-derived address) — that iteration was rejected and the check moved; the incident also surfaced the latent stock bug filed as DF-2782.

fix_status: fixed (fix_baseline_reproduced=1, fix_patched_reproduced=0).

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

With kern.mqueue.mq_max_queues=8, strand exhaust stops after exactly 8 queues / 64.2 MB (mq_open ENOMEM at #9) and a second strander cannot create one queue; sysctl restored to 256. First fix iteration (cap check before lock acquisition, goto exit with mq==NULL) panicked the kernel - rejected, corrected placement; that incident surfaced DF-2782.

fix_validation.log; fixiter1 analysis in VERDICT.md
↓ fix.diffDragonFly dfbsd 6.5-DEVELOPMENT #1: Tue Sep 1 04:57:57 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64

Confirmed kernel references

Detail

Exploit chain

loop: mq_open(unique, {maxmsg=512, msgsize=16352}) -> send 512 x 16 KB messages (8.4 MB/queue) -> close (queue+messages persist) -> exit strander -> kernel retains everything; repeat with new names until kmalloc ENOMEM; no reclamation path without the (withheld) names.

Evidence (decisive lines)

["run.log: 96.4 MB stranded then 'mqueues 7.54K 97.5M' AFTER process exit", "run.exhaust.log: 'mq_send #280: Cannot allocate memory' + 'mq_open: Cannot allocate memory' at ~390 MB; zero progress across 2.5 min of retries", 'fix_validation.log: with kern.mqueue.mq_max_queues=8 the strander stops after exactly 8 queues / 64.2 MB']

PoC changes

msgsize lowered 16384->16352 (the send-path cap includes the 32-byte mq_msg header, so attr-accepted 16384 always EMSGSIZEs on send); brief/exhaust modes; byte accounting includes header+slack.

Verified recommended fix

Add system-wide (and ideally per-uid) queue-count and queued-byte caps as sysctls (kern.mqueue.mq_max_queues in fix.diff, default 256); upstream should also consider an enumeration/cleanup interface.

Verdict

Unprivileged persistent kernel-memory exhaustion proven: 12 queues x 512 msgs x 16 KB = 96.4 MB remained allocated (vmstat -m: mqueues 97.5M) after the strander exited - queues and messages survive last close unless unlinked (sys_mqueue.c:391-399) and cannot be enumerated; sustained stranding drove the kernel to ENOMEM on mq_send/mq_open (M_WAITOK kmalloc failing) at ~390 MB on the 4 GB guest with 2.48 GB user memory still free, and retry loops made zero further progress - the ceiling is kernel-heap-side and the consumption is permanent until reboot. No system-wide or per-user queue/byte accounting exists (only per-process open count and per-queue geometry, sys_mqueue.c:71-76); Linux ships RLIMIT_MSGQUEUE + fs.mqueue.* for exactly this attack class.