# DF-5561 — VERDICT

**Status: reproduced** (deterministic, stock `X86_64_GENERIC` snapshot kernel,
first clean attempt; guest `DragonFly 6.5-DEVELOPMENT #0 Thu Jul 2 06:02:54`).
**Impact: dos** — permanent whole-machine wedge via leaked `db_active`; every
trap in the system (including ordinary user-mode page faults and ASTs) is
rerouted into `trap_fatal()` + `kdb_trap()`, cascading into
`panic: stop_cpus() failed` with all CPUs stopped at a `db>` prompt that
answers to no one. No unprivileged-only entry path was needed for the repro
(root sysctl is the *vehicle*); the bug itself lives in the ddb re-entrancy
glue and bites any real debugging session on a panicking/corrupted box.
**Confidence: certain** (counter read directly in-kernel before/after; full
serial transcripts; patched-kernel differential).
**Fix: validated** (`fix_status=fixed`) — with the one-line guard, the same
trigger sequence leaves `db_active` at 1 through the nested session, and the
post-continue trap storm / stop_cpus panic do not occur.

## Root cause (line-accurate)

`kdb_trap()` (`sys/platform/pc64/x86_64/db_interface.c:103`) arms a global
restart point for traps taken *inside* a live debugger session:

```c
170:  setjmp(db_global_jmpbuf);
171:  db_global_jmpbuf_valid = TRUE;
172:  db_active++;                 /* <-- re-executed on every longjmp re-entry */
...
176:  db_trap(type, code);         /* the ddb session */
...
180:  db_active--;                 /* only the SURVIVING invocation ever runs this */
182:  db_global_jmpbuf_valid = FALSE;
```

When a trap hits during a ddb command and it is *not* covered by
`db_nofault` (e.g. ddb's `call`/`db_fncall` at `sys/ddb/db_command.c:515`,
which invokes an arbitrary address with no fault guard), the nested
`kdb_trap()` takes the default case, sees `db_global_jmpbuf_valid`
(db_interface.c:152-153) and `longjmp`s back to line 170. The `setjmp` now
returns non-zero and execution falls into line 172: **`db_active++` runs a
second time**. The outer invocation — the one that performed the original
increment — has been abandoned by the `longjmp`, so its `db_active--`
(line 180) never executes. One nested-trap event therefore leaks exactly +1
on `db_active`, and the leak is permanent: nothing else ever decrements it.

The amplifier is `sys/platform/pc64/x86_64/trap.c:414-427`:

```c
420:  if (db_active && frame->tf_trapno != T_DNA) {
423:          trap_fatal(frame, eva);
425:          goto out2;
```

With `db_active != 0`, **every** trap on the system — user or kernel, e.g. a
demand-paged exec or the AST on a syscall-return path — is diverted from its
normal handler into `trap_fatal()`, which prints the fatal-trap banner and
calls `kdb_trap()` again (trap.c:1096, `(debugger_on_panic || db_active)`),
which stops all other CPUs and waits for console input. On the observed run
this stormed immediately (`kernel: type 7 trap` from a *user-mode* AST,
`Fatal trap 7 … while in user mode, current process = 816`), and because each
re-entry manages `stopped_cpus` masks afresh, the storm ended in
`panic: stop_cpus() failed` (db_interface.c:195) — whose backtrace shows the
bug's signature: `kdb_trap() at kdb_trap+0x38f` **twice** (panic.txt).

Inherited note: this code structure descends from FreeBSD's historic i386
`db_interface.c`; FreeBSD's modern `kdb(4)` (subr_kdb.c) abandoned this
setjmp dance. The vkernel64 twin
(`sys/platform/vkernel64/x86_64/db_interface.c:169-171`) carries the identical
`db_active++`-after-`setjmp` pattern and should get the same guard.

## Reproduction (baseline, stock kernel)

Exact sequence in README.md; full transcript in `serial.baseline.txt`:

1. `sysctl debug.enter_debugger=ddb` (root) → `db>` (db_active 0→1).
2. `print *db_active` → `1`.
3. `call 0x10()` → `Fatal trap 12` → `kernel: type 12 trap, code=10`
   (db_interface.c:138) → longjmp re-entry → new `db>`.
4. `print *db_active` → **`2`** — the double-increment, read in-kernel.
5. `c` → leaked db_active=1 bites on the very next trap:
   `kernel: type 7 trap, code=2`, `Stopped at 0x8008bffdc` (user RIP),
   `print *db_active` → 2 at that (second) session's prompt.
6. `c` again → `panic: stop_cpus() failed` → `Debugger("panic")` → eternal
   `db>` wedge; `vm.sh status` ⇒ down (ssh dead).

## Fix validation (patched kernel, `#1 Wed Oct 7 18:06:33`-series build)

`fix.diff` guards the increment (`if (db_active == 0) db_active++;`) so the
longjmp re-entry no longer re-counts. Applied inside the guest's `/usr/src`
(never to the audit tree), kernel rebuilt with
`make -j6 nativekernel KERNCONF=X86_64_GENERIC`, installed, rebooted:

- Same trigger: nested prompt `print *db_active` → **`1`**
  (was 2) — `serial.patched.txt`.
- After `c`: `CPU3 restarting CPUs: 0x37` / ` restarted`, then **no** trap
  storm, **no** `stop_cpus()` panic — exactly one `kernel: type 12` line in
  the whole log (the nested entry's own print). The baseline's bad behavior
  is gone (`fix_baseline_reproduced=1`, `fix_patched_reproduced=0`).

## Honest caveat — post-ddb-exit freeze on this rig (pre-existing, orthogonal)

On this QEMU/KVM guest, *any* ddb session exit — including a plain
`sysctl debug.enter_debugger=ddb` → `c` with no trigger at all, on the stock
kernel, with `db_active` correctly returning to 0 — leaves userland timers
frozen: a pre-started `while true; do echo HB $(date); sleep 5; done >
/dev/console` heartbeat stops after ` restarted`, getty does not echo, and
new ssh connections time out (verified across stock and patched builds; 6
retry rounds, plus QEMU-monitor observation: vCPUs alive, mostly idle).
This affects the *control* path identically on both kernels, so it cannot
distinguish stock from patched; DF-5561's own signatures (counter=2,
trap storm, stop_cpus panic) are the discriminating evidence and are exactly
what the fix removes. The freeze itself (systimer/cpustop restart or console
poll-mode restore after a ddb exit) is a separate pre-existing defect worth
its own investigation — likely in the ddb exit path
(db_interface.c:174-201 / mp_machdep cpustop rearm), not in the guarded
increment.

Also observed while tracing this class (out-of-file, recorded for the
orchestrator): the `longjmp` out of `trap_fatal()` equally abandons
`trap.c:422-424`'s `++gd->gd_trap_nesting_level` — the decrement never runs
(same leak pattern, different counter, trap.c's file).

## Threat model

Reaching the bug needs a live ddb session plus one non-nofault-protected
fault inside it. Realistic: any kernel panic (default
`debug.debugger_on_panic=1`) drops the operator into ddb on a *corrupted*
box, where the routine first commands (`ps`, `bt`, `call`) fault on smashed
state; one `c` later the "recovered" system converts every user page fault
into a full stop_cpus+console wait — a panic amplified into an unrecoverable
hang with no dump. The same applies to NMI-triggered ddb
(`machdep.ddb_on_nmi=1` default) and `machdep.ddb_on_seg_fault` setups.

## Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest `make -j6 nativekernel KERNCONF=X86_64_GENERIC` completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.