DF-5561 — kdb_trap() longjmp re-entry double-increments db_active; leaked db_active routes every trap into the debugger (permanent all-CPU wedge)

File: sys/platform/pc64/x86_64/db_interface.c Severity: Medium (panic/debug-path state corruption → system-wide persistent DoS) Status: reproduced (stock kernel, deterministic), fix validated on-guest.

What to build / run

No compilation is needed for the trigger itself — it is a ddb console sequence plus one root sysctl as the entry vehicle:

1. Boot the guest with a writable serial console (ddb reads commands from the console; vm.sh's default -serial file: is output-only): serial_vm.sh up + serial_writer.py (host-side rig, both in this pack).
2. sysctl debug.enter_debugger=ddb (root, over ssh) → box stops at db>.
3. At db> (host: echo '<cmd>' > /tmp/opencode/df-5561-ser.in): - print *db_active → 1 - call 0x10() → ddb's call (db_fncall, sys/ddb/db_command.c:515+) raw-jumps to unmapped 0x10 with no db_nofault protection → kernel-mode T_PAGEFLT inside the ddb session. - at the re-entered prompt: print *db_active → 2 (BUG) - c

Expected output (stock kernel) — see serial.baseline.txt

db> print *db_active
       1
db> call 0x10()
Fatal trap 12: page fault while in kernel mode
...
kernel: type 12 trap, code=10          <-- kdb_trap() default case (db_interface.c:138)
Stopped at      Debugger+0x7c: ...
db> print *db_active
       2                               <-- double-increment: the longjmp re-entry
db> c                                  re-executed db_active++ (db_interface.c:172)
kernel: type 7 trap, code=2            <-- leaked db_active: the NEXT trap (an AST
Stopped at      0x8008bffdc: ...       on the syscall-return path!) re-enters ddb
panic: stop_cpus() failed              <-- cascades: two concurrent stop masks
Trace beginning at frame ...
kdb_trap() at kdb_trap+0x38f           <-- the nested re-entry, twice on the stack
kdb_trap() at kdb_trap+0x38f
trap_fatal() at trap_fatal+0x2cd
trap() at trap+0x85
Debugger("panic")

After this the machine sits in ddb forever; ssh no longer answers (Connection timed out during banner exchange, vm.sh status ⇒ down).

With fix.diff applied — see serial.patched.txt

Same sequence: nested prompt shows print *db_active → 1; after c, CPU3 restarting CPUs … restarted and then silence — no trap storm, no stop_cpus() panic. Exactly one kernel: type 12 trap line (the nested entry's own print) vs. the stock cascade.

Note on the post-ddb-exit freeze (control, not part of this bug)

On this QEMU rig, any ddb session exit (even sysctl enter → c with zero triggers, db_active correctly back at 0) leaves userland timers frozen (heartbeat test in VERDICT.md; stock and patched behave identically). That pre-existing exit-path/rig behavior is orthogonal to DF-5561 — the finding's baseline signatures (counter=2, trap storm, stop_cpus panic) are what the fix removes, and they are absent on the patched kernel.

DF-5561 — VERDICT

Status: reproduced (deterministic, stock X86_64_GENERIC snapshot kernel, first clean attempt; guest DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 06:02:54). Impact: dos — permanent whole-machine wedge via leaked db_active; every trap in the system (including ordinary user-mode page faults and ASTs) is rerouted into trap_fatal() + kdb_trap(), cascading into panic: stop_cpus() failed with all CPUs stopped at a db> prompt that answers to no one. No unprivileged-only entry path was needed for the repro (root sysctl is the vehicle); the bug itself lives in the ddb re-entrancy glue and bites any real debugging session on a panicking/corrupted box. Confidence: certain (counter read directly in-kernel before/after; full serial transcripts; patched-kernel differential). Fix: validated (fix_status=fixed) — with the one-line guard, the same trigger sequence leaves db_active at 1 through the nested session, and the post-continue trap storm / stop_cpus panic do not occur.

Root cause (line-accurate)

kdb_trap() (sys/platform/pc64/x86_64/db_interface.c:103) arms a global restart point for traps taken inside a live debugger session:

170:  setjmp(db_global_jmpbuf);
171:  db_global_jmpbuf_valid = TRUE;
172:  db_active++;                 /* <-- re-executed on every longjmp re-entry */
...
176:  db_trap(type, code);         /* the ddb session */
...
180:  db_active--;                 /* only the SURVIVING invocation ever runs this */
182:  db_global_jmpbuf_valid = FALSE;

When a trap hits during a ddb command and it is not covered by db_nofault (e.g. ddb's call/db_fncall at sys/ddb/db_command.c:515, which invokes an arbitrary address with no fault guard), the nested kdb_trap() takes the default case, sees db_global_jmpbuf_valid (db_interface.c:152-153) and longjmps back to line 170. The setjmp now returns non-zero and execution falls into line 172: db_active++ runs a second time. The outer invocation — the one that performed the original increment — has been abandoned by the longjmp, so its db_active-- (line 180) never executes. One nested-trap event therefore leaks exactly +1 on db_active, and the leak is permanent: nothing else ever decrements it.

The amplifier is sys/platform/pc64/x86_64/trap.c:414-427:

420:  if (db_active && frame->tf_trapno != T_DNA) {
423:          trap_fatal(frame, eva);
425:          goto out2;

With db_active != 0, every trap on the system — user or kernel, e.g. a demand-paged exec or the AST on a syscall-return path — is diverted from its normal handler into trap_fatal(), which prints the fatal-trap banner and calls kdb_trap() again (trap.c:1096, (debugger_on_panic || db_active)), which stops all other CPUs and waits for console input. On the observed run this stormed immediately (kernel: type 7 trap from a user-mode AST, Fatal trap 7 … while in user mode, current process = 816), and because each re-entry manages stopped_cpus masks afresh, the storm ended in panic: stop_cpus() failed (db_interface.c:195) — whose backtrace shows the bug's signature: kdb_trap() at kdb_trap+0x38f twice (panic.txt).

Inherited note: this code structure descends from FreeBSD's historic i386 db_interface.c; FreeBSD's modern kdb(4) (subr_kdb.c) abandoned this setjmp dance. The vkernel64 twin (sys/platform/vkernel64/x86_64/db_interface.c:169-171) carries the identical db_active++-after-setjmp pattern and should get the same guard.

Reproduction (baseline, stock kernel)

Exact sequence in README.md; full transcript in serial.baseline.txt:

1. sysctl debug.enter_debugger=ddb (root) → db> (db_active 0→1).
2. print *db_active → 1.
3. call 0x10() → Fatal trap 12 → kernel: type 12 trap, code=10 (db_interface.c:138) → longjmp re-entry → new db>.
4. print *db_active → 2 — the double-increment, read in-kernel.
5. c → leaked db_active=1 bites on the very next trap: kernel: type 7 trap, code=2, Stopped at 0x8008bffdc (user RIP), print *db_active → 2 at that (second) session's prompt.
6. c again → panic: stop_cpus() failed → Debugger("panic") → eternal db> wedge; vm.sh status ⇒ down (ssh dead).

Fix validation (patched kernel, #1 Wed Oct 7 18:06:33-series build)

fix.diff guards the increment (if (db_active == 0) db_active++;) so the longjmp re-entry no longer re-counts. Applied inside the guest's /usr/src (never to the audit tree), kernel rebuilt with make -j6 nativekernel KERNCONF=X86_64_GENERIC, installed, rebooted:

• Same trigger: nested prompt print *db_active → 1 (was 2) — serial.patched.txt.
• After c: CPU3 restarting CPUs: 0x37 / restarted, then no trap storm, no stop_cpus() panic — exactly one kernel: type 12 line in the whole log (the nested entry's own print). The baseline's bad behavior is gone (fix_baseline_reproduced=1, fix_patched_reproduced=0).

Honest caveat — post-ddb-exit freeze on this rig (pre-existing, orthogonal)

On this QEMU/KVM guest, any ddb session exit — including a plain sysctl debug.enter_debugger=ddb → c with no trigger at all, on the stock kernel, with db_active correctly returning to 0 — leaves userland timers frozen: a pre-started while true; do echo HB $(date); sleep 5; done > /dev/console heartbeat stops after restarted, getty does not echo, and new ssh connections time out (verified across stock and patched builds; 6 retry rounds, plus QEMU-monitor observation: vCPUs alive, mostly idle). This affects the control path identically on both kernels, so it cannot distinguish stock from patched; DF-5561's own signatures (counter=2, trap storm, stop_cpus panic) are the discriminating evidence and are exactly what the fix removes. The freeze itself (systimer/cpustop restart or console poll-mode restore after a ddb exit) is a separate pre-existing defect worth its own investigation — likely in the ddb exit path (db_interface.c:174-201 / mp_machdep cpustop rearm), not in the guarded increment.

Also observed while tracing this class (out-of-file, recorded for the orchestrator): the longjmp out of trap_fatal() equally abandons trap.c:422-424's ++gd->gd_trap_nesting_level — the decrement never runs (same leak pattern, different counter, trap.c's file).

Threat model

Reaching the bug needs a live ddb session plus one non-nofault-protected fault inside it. Realistic: any kernel panic (default debug.debugger_on_panic=1) drops the operator into ddb on a corrupted box, where the routine first commands (ps, bt, call) fault on smashed state; one c later the "recovered" system converts every user page fault into a full stop_cpus+console wait — a panic amplified into an unrecoverable hang with no dump. The same applies to NMI-triggered ddb (machdep.ddb_on_nmi=1 default) and machdep.ddb_on_seg_fault setups.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.