DF-5722 — Resuming DDB entered from interrupt context panics the kernel

Severity: Low (root-console trigger; deterministic kernel panic — any attempt to debug interrupt-path code with breakpoints crashes the box)

Setting a DDB breakpoint in any code that runs in interrupt context (e.g. b hardclock) and then continuing (c) deterministically panics on resume:

panic: td_critcount is/would-go negative! 0xfffff8008f313f00 -1
cpuid = 0
Trace beginning at frame 0xfffff801154ab548
crit_panic() at crit_panic+0x2f 0xffffffff80672cff
crit_panic() at crit_panic+0x2f 0xffffffff80672cff
crit_exit_wrapper() at crit_exit_wrapper+0x3c 0xffffffff806734ac

Reproduced 3/3 boots (with variations: + software watchpoint, + invisible- step roundtrip, and the minimal single-session recipe). Thread-context DDB sessions on the same kernel resume cleanly (multiple controls, incl. a breakpoint hit in sys_kill resumed with c → machine alive), so the trigger is specifically the interrupt-context session.

Minimal recipe (run4-minimal-panic.serial.log)

db> b hardclock
db> c
CPU0 restarting CPUs: 0x0000000000000000
 restarted
CPU0 stopping CPUs: 0x00000000
 stopped
Breakpoint at   hardclock:      pushq   %rbp
db> d hardclock
db> c
CPU0 restarting CPUs: 0x0000000000000000
 restarted
panic: td_critcount is/would-go negative! 0xfffff8008f313f00 -1

One DDB session, entered from the clock-interrupt path via the INT3, no step-over, no watchpoint, no other state. The panic fires as the session exits (crit_exit() at db_interface.c:201 → crit_exit_wrapper() → _crit_exit → crit_panic, sys/kern/lwkt_thread.c:1766-1781): the thread's td_critcount was decremented once too often across the session.

Bisect matrix (all on the same stock kernel)

db_run.c's engine executes identical, context-agnostic code in every row (state captured live: db_run_mode transitions, rfl TF-clear at every stop, DR registers idle) — the defect is in the machine layer's crit accounting across kdb_trap() for interrupt-context sessions (sys/platform/pc64/x86_64/db_interface.c:160-201 crit_enter/crit_exit, in interaction with the interrupt entry/exit paths' TD_CRITCOUNT bookkeeping, ipl.s/exception.S/interrupt stubs).

Filed from the sys/ddb/db_run.c pass-2 audit (where it was discovered and characterized via db_run.c's step/continue/breakpoint machinery); the fix owner is db_interface.c / ipl.s.

Threat model

Requires a DDB breakpoint in interrupt-context code — console/root only (unprivileged reachability probes all negative, see DF-5721 pack). Not an escalation; it is a debugger-robustness defect: any root operator who sets a breakpoint in a clock/interrupt/ITHREAD path gets a guaranteed panic + crash dump + reboot instead of a debugging session. Three crash dump sets were taken on the guest (/var/crash info.0-2, kern.0-2, vmcore.0-2) for offline analysis.

Files

• run4-minimal-panic.serial.log — minimal recipe + panic
• run5-threadctx-control.serial.log — thread-context controls (alive)
• up-panics.fullserial.log — full serial history (all runs on final boot)
• panic.watch.serial.log — run1 (watch + interrupt sessions) panic
• fix.diff — defensive kdb_trap critcount save/restore
• verdict.json / manifest.json

Reproduce

1. ./../DF-5721/serial_vm1.sh up (UP guest — on SMP guests the resume path wedges earlier in stop_cpus/restart_cpus, DF-5720 lead, and the recipe does not complete)
2. ssh -F dfbsd-qemu/config dfbsd 'sysctl debug.enter_debugger=ddb' &
3. On serial: b hardclock, c, wait for "Breakpoint at hardclock", then d hardclock, c → panic + dump + auto-reboot.

DF-5722 — VERDICT

Status: reproduced (deterministic, 3/3 boots, stock X86_64_GENERIC snapshot kernel on a 1-vCPU QEMU/KVM guest with socket serial console). Impact: panic (root-console-triggerable kernel panic; debugger robustness defect, not privilege escalation). Confidence: certain for the trigger and the layer attribution; the exact leak site inside the session window is narrowed to db_interface.c:160-201 + interrupt entry/exit bookkeeping and would need dump analysis to pin to a single instruction (three vmcores are preserved on the guest /var/crash).

Narrative

During the pass-2 audit of sys/ddb/db_run.c I live-verified the whole step/continue engine on a UP guest (thread-context sessions all clean: TF set/clear symmetry — rfl=0x246 TF-clear at every stop after s; breakpoint dispatch and count handling — "Breakpoint at" banners, count reset, STEP_INVISIBLE invisible-step roundtrips; continue paths). While verifying the breakpoint dispatch in interrupt context (b hardclock), every resume panicked:

panic: td_critcount is/would-go negative! <td> -1 crit_panic ← crit_panic ← crit_exit_wrapper (sys/kern/lwkt_thread.c:1772-1781, called from crit_exit() at sys/platform/pc64/x86_64/db_interface.c:201 — the C wrapper, the only crit_exit_wrapper call site on this path)

Bisect (full matrix in README): - thread-context sessions: sysctl-entry, single-step, breakpoint-hit in sys_kill resumed with c — ALL alive. - interrupt-context session (breakpoint in hardclock, the clock path): PANIC on resume — minimal recipe = b hardclock, c, d hardclock, c (run4-minimal-panic.serial.log). - The DF-5721 software-watchpoint was exonerated by run #3 (panic without any watchpoint). - The SMP-guest starvation (DF-5720 lead) is a separate failure: on the SMP guest the second kdb_trap entry's stop_cpus hangs before any resume completes; on UP (stop_cpus no-op) the resume completes and the interrupt-context panic is what surfaces. Both live-captured; neither is db_run.c (see below).

Why db_run.c is exonerated (this audit's file)

• The engine is context-agnostic: db_stop_at_pc/db_restart_at_pc ran identical code in the surviving thread-context controls and the panicking interrupt-context runs; captured state identical (db_run_mode 0/4/5 transitions per code, TF clear at stops, DRs idle, breakpoint list operations symmetric).
• The panic fires in the machine layer's critical-section accounting around/inside kdb_trap (db_interface.c:160-201) in interaction with interrupt entry/exit TD_CRITCOUNT bookkeeping (ipl.s doreti/doreti_soft temporary decls at ipl.s:281-283/304-306, interrupt stub guarded inc/dec msi_vector.s:56-79): across an interrupt-context session one td_critcount decrement goes unmatched, and the session-exit crit_exit() underflows (0 → -1 → crit_panic).

Impact

Root console only (all unprivileged entry routes closed — probes in the DF-5721 pack: user int3 → SIGTRAP; PT_STEP artifact filtered; PT_SETDBREGS kernel-addr EINVAL; sysctl door EPERM). Consequence for the privileged operator: any breakpoint in interrupt-context code (clock, ITHREAD, driver interrupt paths — the bread and butter of kernel debugging) converts the first c into a panic + dump + reboot. Local DoS requires root, so: Low severity, panic/dos impact class.

Fix

fix.diff (defensive, authored against the read-only sys/ tree): kdb_trap saves curthread->td_critcount after its crit_enter() and force-restores it before returning (replacing the bare crit_exit() at db_interface.c:201). This makes the debugger session transparent to the interrupted context's crit accounting regardless of where inside the session the count leaked, and directly prevents the observed would-go negative exit. NOT validated by rebuild+rerun: the panic recipe requires a UP guest (SMP guests wedge in stop_cpus first, a different defect) and a kernel rebuild on the 1-vCPU guest was not feasible in-run. fix_status=not_testable. The upstream-quality fix is to find the unmatched decrement (dump analysis of the preserved vmcores); the save/restore is the safe belt that stays correct even then.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.