β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-2782

sys_mq_open race-gated NULL-pointer lockmgr panic: the under-lock EMFILE branch jumps to the shared exit label while mq is still NULL, releasing a lock on a NULL-derived address

Field Value
ID DF-2782
Status new
Severity Medium
CVSS 3.1 CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:N/I:N/A:H
CWE CWE-476 NULL Pointer Dereference
File sys/kern/sys_mqueue.c
Lines 577-580 (goto with mq NULL), 595-596 (shared release)
Area kern
Confidence certain
Discovered 2026-08-31
Pass 2 (GLM 5.3 second pass β€” surfaced during Phase V of this run)
Bucket memcorrupt
Reported pending
Known CVE none
CVE match novel

Summary

In sys_mq_open(), when the queue is not found and O_CREAT is set, the under-lock limit check executes error = EMFILE; goto exit; while mq is still NULL (mq = mq_new happens only after the check). The shared exit label then runs lockmgr(&mq->mq_mtx, LK_RELEASE) on &((struct mqueue*)NULL)->mq_mtx (~0x100) β€” guaranteed page fault. Reachable whenever p_mqueue_cnt reaches mq_open_max between the early unlocked check and the under-lock check: arranged with 8 barrier-synchronized threads racing from a count of 511. INVARIANTS- independent; found during Phase V of this run.

Proof of contest

VERIFIED on the stock guest (findings/poc/DF-2782/limit_race.c): seed 511, barrier-release 8 threads into mq_open(O_CREAT) β†’ Fatal trap 12 ... Stopped at lockmgr_release+0x11, guest down (panic.txt). Fix (guard the shared release with if (mq)) in the pack.

 exit:
-   lockmgr(&mq->mq_mtx, LK_RELEASE);
+   if (mq)
+       lockmgr(&mq->mq_mtx, LK_RELEASE);
    lockmgr(&mqlist_mtx, LK_RELEASE);

Timeline

  • 2026-08-31 Discovered during pass-2 verification of sys_mqueue.c (GLM 5.3); unpriv race panic reproduced same run.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-2782 Β· 13 files
FileTypeDescriptionSize
limit_race.c β€” 3.3 KB view raw
build.sh β€” 53 B view raw
run.sh β€” 166 B view raw
build.log β€” 36 B view raw
run.log β€” 1.1 KB view raw
run.ssh.log β€” 48 B view raw
panic.txt β€” 2.1 KB view raw
fixiter1_context.txt β€” 2.1 KB view raw
fix_validation.log β€” 129 B view raw
VERDICT.md β€” 1.9 KB ↓ raw
verdict.json β€” 2.9 KB view raw
fix.diff β€” 406 B view raw
env.txt β€” 119 B view raw
VERDICT.md
↓ download raw

VERDICT β€” DF-2782: sys_mq_open NULL-pointer lockmgr panic (race-gated limit path)

Status: REPRODUCED Β· impact: panic Β· confidence: certain

What was proven on the guest (stock kernel, build #0)

/tmp/limit_race 300 (8 threads, barrier-synchronized from a count of 511): kernel panicked mid-run β€” serial console (panic.txt):

Fatal trap 12: page fault while in kernel mode
current process = 842
kernel: type 12 trap, code=0
Stopped at      lockmgr_release+0x11:  movq 0x8(%rdi),%rsi

ssh dropped mid-run (Connection ... closed by remote host), vm.sh status β‡’ down β€” the honest panic signature.

Root cause path: two (or more) threads pass the early unlocked limit check at sys_mqueue.c:452 while p_mqueue_cnt == 511; the first completes its open (count 512); the second reaches the under-lock check at :577, takes the error = EMFILE; goto exit; branch β€” but mq is still NULL in this branch (mq = mqueue_lookup(name) returned NULL; mq = mq_new happens only at :583, AFTER the check) β€” and the shared exit label at :595-596 executes lockmgr(&mq->mq_mtx, LK_RELEASE) on a NULL-derived address (~0x100) β†’ page fault β†’ panic.

Corroboration: the identical crash signature was independently produced when a rejected first-version DF-2781 fix diff introduced a goto exit with mq == NULL at the same site (fixiter1_context.txt) β€” same trap, same instruction β€” confirming the exit-path shape is the hazard and that the stock :577 check has exactly that shape.

Fix validation (patched kernel, build #1 04:57)

fix.diff: guard the shared release (if (mq)) β€” mq is non-NULL on every other path reaching the label. Patched kernel: the same PoC ran 300 rounds Γ— 8 threads ("round 299 survived ... survived 300 rounds without the panic"), guest up (fix_validation.log).

fix_status: fixed (fix_baseline_reproduced=1, fix_patched_reproduced=0).

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

Patched kernel (if (mq) guard): the same PoC ran 300 rounds x 8 threads without a panic ('survived 300 rounds'), guest up.

fix_validation.log
↓ fix.diffDragonFly dfbsd 6.5-DEVELOPMENT #1: Tue Sep 1 04:57:57 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64

Confirmed kernel references

Detail

Exploit chain

seed p_mqueue_cnt to 511 -> barrier-release N threads into mq_open(O_CREAT) -> >=2 threads pass :452 at 511 -> first increments to 512 -> second hits :577 EMFILE -> goto exit with mq==NULL -> lockmgr on ~&((struct mqueue*)NULL)->mq_mtx (offset ~0x100) -> page fault -> panic. INVARIANTS-independent.

Evidence (decisive lines)

['panic.txt: Fatal trap 12, Stopped at lockmgr_release+0x11, current process 842, db> prompt', 'run.log: ssh dropped mid-run + vm status down', 'fixiter1_context.txt: identical signature via mq==NULL goto-exit', 'fix_validation.log: 300 rounds survived on patched kernel']

PoC changes

none vs plan (unique per-iteration queue names to force the CREATE path, barrier+seed-511 choreography; round cleanup of fds and unlinks).

Verified recommended fix

Guard the shared exit-path release with if (mq) (fix.diff) - mq is non-NULL on every other path reaching the label.

Verdict

Race-gated NULL-pointer lockmgr panic in sys_mq_open proven on the stock kernel: 8 barrier-synchronized threads racing past the early unlocked limit check (sys_mqueue.c:452) from p_mqueue_cnt=511; when the counter reaches 512 in-flight, a thread takes the under-lock EMFILE branch at :577-580 and goto-exits while mq is still NULL (mq=mq_new happens only at :583), so the shared exit label (:595-596) executes lockmgr(&mq->mq_mtx, LK_RELEASE) on a NULL-derived address - kernel panicked: 'Fatal trap 12 ... Stopped at lockmgr_release+0x11', guest down. Independently corroborated: an identical signature was produced by a rejected fix iteration that introduced the same mq==NULL goto-exit shape at the same label (fixiter1_context.txt).