exc_segfault() tail splz() is unguarded: deferred IPIs/interrupt work dispatched inside the interrupted context's critical section and inside outer interrupt handlers β protocol divergence from pc64 doreti's explicit TD_CRITCOUNT gate
Summary
splz() runs unconditionally at whatever critcount/nesting the fault interrupted, while pc64 doreti refuses to unpend interrupts during a critical section (ipl.s:132-146) and every other splz call site goes through guarded wrappers (lwkt_maybe_splz: critcount==0 AND nesting==0 AND nest_count<2). A deferred IPI (pmap shootdown, lwkt_send_msg) can interleave mid-critical-section on guest-kernel data whose only lock is the critical section - torn reads/lost updates inside the VM; DDB single-stepping makes it deterministic. Model reproducer: INVARIANT_BROKEN with callback_critcount=1 + torn state 3/3 (pc64-gate control 3/3 OK); in-vkernel 2000-fault storm validates the rig. Trap-side sibling is DF-5338. critcount/nesting/nest_count-gate fix proposed; leads TF_R10/R11 (exception.c side KILLED - the defect is pc64 exception.S Xfast_syscall:402-427, cross-file) and SA_NODEFER-storm (KILLED with runtime evidence) both resolved.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-5586 Β· 14 files| File | Type | Description | Size | |
|---|---|---|---|---|
| splzcrit.c | β | 7.8 KB | view raw | |
| storm.c | β | 5.1 KB | view raw | |
| build.sh | β | 66 B | view raw | |
| run.sh | β | 163 B | view raw | |
| build_image.sh | β | 1.2 KB | view raw | |
| run_vk.sh | β | 1.6 KB | view raw | |
| build_and_model.log | β | 868 B | view raw | |
| run_vk.log | β | 940 B | view raw | |
| fix.diff | β | 818 B | view raw | |
| VERDICT.md | β | 5.5 KB | β raw | |
| README.md | β | 1.8 KB | β raw | |
| verdict.json | β | 5.0 KB | view raw | |
| manifest.json | β | 1.1 KB | view raw | |
| env.txt | β | 526 B | view raw |
DF-5586 β PoC pack
Build (on the DragonFly guest)
sh build.sh # cc -static -O2 -o splzcrit splzcrit.c
# cc -static -O0 -o storm storm.c
Run
1. Dispatch-violation model (no root needed)
sh run.sh # ./splzcrit vkernel ; ./splzcrit pc64
Expected:
== vkernel mode == mode=vkernel victim.lo=1 victim.hi=2 callback_critcount=1 callback_torn=1 INVARIANT_BROKEN (IPI callback ran inside the critical section ...) RC_VKERNEL=1 == pc64 mode == mode=pc64 victim.lo=1 victim.hi=1 callback_critcount=0 callback_torn=0 INVARIANT_OK RC_PC64=0
2. In-vkernel fault storm (root; needs the VKERNEL64 build + image)
sh build_image.sh # creates /vkroot.img with /root/storm (DF-5474
# link workaround must be in
# sys/platform/vkernel64/conf/Makefile; see
# findings/poc/DF-5474)
sh run_vk.sh # boots kernel.debug -m 512m -r /vkroot.img -n 2,
# runs "/root/storm 500 1" on the guest console
Expected (console): STORM_DONE iters=500 intdiv=500 ill=500 prot=500
map=500 + VKERNEL_SURVIVED_IF_YOU_SEE_THIS + STORM_RC=0, vkernel
ALIVE at the end. This is the runtime kill of the "SA_NODEFER fault
storms" lead: guest-user storms never reach exc_segfault.
Files
splzcrit.cβ model reproducer (vkernel vs pc64 modes)storm.cβ guest-userland fault-storm binary for the vkernelbuild.sh/run.shβ model build/runbuild_image.sh/run_vk.shβ vkernel image + boot/storm driverbuild_and_model.logβ full build + 3Γ model runs (untrimmed)run_vk.logβ vkernel storm console (untrimmed tail)fix.diffβ guard the tail splz() (doreti-equivalent)verdict.json,manifest.json,VERDICT.md
DF-5586 β vkernel exc_segfault() tail splz() is unguarded: deferred IPIs / interrupt work dispatched inside the interrupted context's critical section (protocol divergence from pc64 doreti's explicit TD_CRITCOUNT gate)
What
sys/platform/vkernel64/x86_64/exception.c:269-271 β the vkernel's
kernel-mode fault return path:
save = errno;
kern_trap((struct trapframe *)&ctx->uc_mcontext.mc_rdi);
splz(); /* β unconditional */
errno = save;
The real kernel runs pending interrupts on trap return through doreti,
which refuses to dispatch anything while the interrupted thread holds a
critical section β sys/platform/pc64/x86_64/ipl.s:132-146:
* If we are in a critical section we cannot run any pending ints.
...
testl $-1,TD_CRITCOUNT(%rbx) /* can't unpend if in critical sec */
jne 5f /* β straight to doreti_iret */
and the generic kernel only ever invokes splz() through guarded wrappers
(lwkt_maybe_splz(), sys/kern/lwkt_thread.c:1091-1101: critcount==0 β§
gd_intr_nesting_level==0 β§ td_nest_count<2; splz_check(), :1068-1078).
The vkernel's own splz() (sys/platform/vkernel64/platform/machintr.c:122-158)
has no guard of any kind, and exc_segfault is the only caller that
invokes it at the interrupted context's arbitrary crit/nesting state
(go_user's loop, cpu_idle, and crit_exit all dispatch from base
level).
Consequence: when a fault is taken while the interrupted vkernel-kernel
context (a) holds a critical section with a deferred IPI pending
(ipisig deferred it via need_ipiq(), exception.c:92), or (b) is nested
inside another interrupt's signal (ipisig/kqueuesig/timersig all run
with gd_intr_nesting_level raised and empty sa_mask), the tail splz()
executes IPI callbacks (lwkt_process_ipiq), soft/hard interrupt threads
(sched_ithd_soft / sched_ithd_hard_virtual), vktimer_intr, and
kqueue_intr inside the critical section / inside the outer interrupt
handler β precisely the interleaving DragonFly critical sections exist to
prevent. No TD_NEST_COUNT accounting is done anywhere on this path (pc64
bumps it around exactly these dispatches, ipl.s:280-286).
Reachable fault sources at kernel-mode crit/nesting (no vkernel bug
required for the class, though frequency is timing-dependent):
- guest-KVA demand VPTE installs on the MAP_VPAGETABLE window
(init.c:608-624) β the vkernel's normal kernel page-fault mechanism β
occurring inside any of the (ubiquitous) vkernel-kernel critical
sections while a cross-CPU IPI (guest-userland TLB-shootdown churn) is
deferred;
- single-step / breakpoint SIGTRAPs during vkernel DDB sessions
(kern_trap T_TRCTRAP/T_BPTFLT, trap.c:716-725; the spurious-SIGTRAP
comment at trap.c:732-742 acknowledges these arrive in kernel mode);
- any recoverable fault inside IPI/kqueue/timer callback execution.
Impact class: lost critical-section atomicity on guest-kernel data (torn reads/lost updates on structures whose only lock is the critical section) β guest-kernel memory corruption inside the VM (panic or worse). The sibling fall-through defect of the same file family is DF-5338 (trap.c); this finding is the dispatch-side divergence in exception.c.
Reproducer
splzcrit.c β faithful element-for-element model of the dispatch
structure on the DragonFly host: SA_NODEFER/empty-mask fault handler with
a recoverable fault (mmap-in-handler = trap_pfault/vm_fault SUCCESS) and
tail splz (vkernel mode) vs the doreti TD_CRITCOUNT gate (pc64 mode);
deferred-SIGUSR1 "IPI" callback that reads/writes crit-protected state
mid-section.
- vkernel mode:
INVARIANT_BROKEN,callback_critcount=1,callback_torn=1β the IPI callback ran inside the critical section and observed torn state (3/3 runs). - pc64 mode:
INVARIANT_OK,callback_critcount=0, no torn state (3/3 runs).
storm.c + build_image.sh + run_vk.sh β in-vkernel fault storm on a
real booted VKERNEL64 (-n 2): 500 iterations Γ 4 guest-user fault classes
(intdiv/SIGFPE, ud2/SIGILL, RO-write/SIGSEGV, PROT_NONE/SIGSEGV) plus a
shared-memory mprotect-churn sibling driving cross-CPU TLB-shootdown IPIs
for the whole storm. Result: STORM_DONE ... STORM_RC=0, vkernel ALIVE
β normal guest-user fault storms never reach exc_segfault (they return
via vmspace_ctl β go_user β user_trap, trap.c:1407-1439), which is the
runtime kill of the "SA_NODEFER under fault storms" lead (with DF-5338
owning the one deterministic recursion instance).
Fix
Guard the tail splz() exactly like doreti/lwkt_maybe_splz (fix.diff,
applied and rebuilt on the guest; patched vkernel passes the identical
storm; pc64-mode model run demonstrates the guarded semantics).
Verdict summary
- status: reproduced (dispatch violation + torn-state corruption: deterministic 3/3 in the faithful model on the DragonFly host; code divergence proven line-by-line against the real kernel's own gate)
- demonstrated impact ceiling in the live vkernel: none observed (needs a kernel-side recoverable fault to coincide with crit+deferred IPI; DDB single-step makes it deterministic under debugging) β severity Low, bucket memcorrupt, confidence likely.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedfix.diff applied to the guest /usr/src (combined with DF-5587's guard), VKERNEL64 rebuilt cleanly, identical in-vkernel fault storm passes (STORM_DONE, STORM_RC=0, ALIVE) β no regression; the guard's dispatch semantics are exactly the pc64 mode of the model reproducer (callback_critcount=0, INVARIANT_OK, deferred to crit_exit).
findings/poc/DF-5586/run_vk.log (baseline storm) + patched-vkernel run (STORM_RC=0 ALIVE) + build_and_model.log pc64-mode section; fix.diff
Confirmed kernel references
Detail
Evidence (decisive lines)
findings/poc/DF-5586/build_and_model.log (3x model runs: vkernel mode INVARIANT_BROKEN callback_critcount=1 callback_torn=1 vs pc64 mode INVARIANT_OK callback_critcount=0); run_vk.log (in-vkernel storm STORM_DONE iters=500 intdiv=500 ill=500 prot=500 map=500, STORM_RC=0, vkernel ALIVE); fix validation: patched VKERNEL64 (fix.diff applied, rebuilt) passes the identical storm (STORM_RC=0, ALIVE) and pc64-mode model run demonstrates the guarded semantics; code divergence: exception.c:269-271 vs ipl.s:132-146, lwkt_thread.c:1068-1101, machintr.c:122-158
PoC changes
Authored from scratch (no seed): faithful extraction of exception.c's dispatch structure (SA_NODEFER/empty-mask install, deferred-SIGUSR1 ipisig model, recoverable SIGSEGV via mmap-in-handler, splz tail) with a pc64-doreti-gate comparison mode; storm.c needed -O0 (gcc -O2 loop rotation misplaces &&label resume targets) and small iteration counts (~0.1ms per fault+sigreturn cycle); storm argv parsing order fixed; vkernel rig rebuilt from the DF-5556 recipe (DF-5474 link workaround, /var/vkernel dir required).
Verified recommended fix
Gate the tail splz() like doreti/lwkt_maybe_splz: only when curthread->td_critcount==0 && mycpu->gd_intr_nesting_level==0 && curthread->td_nest_count<2
Verdict
exc_segfault's tail splz() (exception.c:270) dispatches deferred IPIs/interrupt work at the interrupted context's critcount/nesting state, violating the dispatch protocol the real kernel enforces (pc64 doreti's explicit TD_CRITCOUNT gate at ipl.s:143-146 and lwkt_maybe_splz's critcount==0/nesting==0/nest_count<2 guards at lwkt_thread.c:1091-1101); the vkernel splz (machintr.c:122-158) has no guards. Reproduced deterministically 3/3 in a faithful signal-structure model on the DragonFly guest: with the vkernel's unguarded tail, the deferred IPI callback executes inside the interrupted critical section and observes/creates torn state (callback_critcount=1, callback_torn=1, INVARIANT_BROKEN lo=1/hi=2); with the pc64 doreti gate, dispatch is deferred to the crit_exit transition (callback_critcount=0, INVARIANT_OK). In the live vkernel the trigger needs a kernel-mode fault (guest-KVA demand VPTE install, or DDB single-step SIGTRAPs) to coincide with a held critical section and a deferred IPI β code-certain, frequency timing-dependent, so demonstrated in-vkernel impact is none; the in-vkernel fault storm (2000 guest-user faults + cross-CPU IPI churn on a 2-vCPU VKERNEL64) confirms normal guest-user fault storms never reach exc_segfault (they return via vmspace_ctl -> go_user -> user_trap), killing the pass-1 'SA_NODEFER under fault storms' concern, whose only deterministic recursion instance is already filed as DF-5338.
No comments yet.