# DF-2712 — verification verdict

## Bottom line

**The claimed whole-system wedge did NOT reproduce.** The *mechanism* it is
built on is real and was reproduced line-accurately — 238,205 spin-held
`bsd4_kick_helper` executions and 48 spin-held FIFO-full
`lwkt_send_ipiq3()` liveloops (interrupts enabled, spinning, while the
global `bsd4_spin` is held) — but on this 6-vCPU guest every liveloop
drained in under a second, the outgoing FIFO depth never exceeded 20 of the
128 entries needed to *enter* the liveloop as a sustained state, and the
scheduler-spin watchdog never observed a frozen cpu. Alongside the
transient liveloops the unprivileged storm produced minutes-scale ssh
starvation and load-average 30-71 — a severe *recoverable* availability
degradation, not the claimed permanent hang. An honest inconclusive for the
wedge claim, with the gap quantified below.

## Line-verified mechanism (all confirmed on the running kernel)

1. `bsd4_kick_helper` (`sys/kern/usched_bsd4.c:1590-1615`) executes
   `lwkt_send_ipiq` (:1611) / `wakeup` (:1613) while the caller holds
   `bsd4_spin` (its own comment :1586 admits it). Call sites inside
   `bsd4_chooseproc_locked_cache_coherent` at :1473/:1512/:1519/:1561,
   which runs under the spin taken at :533 (`bsd4_select_curproc`).
2. That path is LIVE under bsd4 by default: `usched_bsd4_cache_coherent`
   is set to 1 at registration (`usched_bsd4.c:2030`) whenever the cpu
   topology provides nodes (true on this QEMU guest: `usched_bsd4.smt:
   NOT SUPPORTED` + `cache_coherent: 1`).
3. `lwkt_send_ipiq3` (`sys/kern/lwkt_ipiq.c:181`): with
   `gd_processing_ipiq == 0` the liveloop threshold is **MAXCPUFIFO/2 =
   128** (:225), not 253 (253 is the nested case, :228). On `> 128`
   outstanding it does `cpu_enable_intr()` (:240) and liveloops (:244-277)
   until the target drains to <= 64 — still holding `bsd4_spin`.
4. **Reproduced**: `lwkt.ipiq_fifofull` rose 0 -> 48 across the unpriv
   storms (each entry = one spin-held liveloop). All exited < 1 s: zero
   `send_ipiq %d->%d tgt not draining` console lines across every run
   (that kprintf fires after 1 s stuck, lwkt_ipiq.c:258-275).
5. Unprivileged reachability confirmed: `lwp_setaffinity(2)` with pid=0 is
   explicitly allowed for self (`kern_usched.c:423` "Always allow change
   self CPU affinity"; `usched_set(USCHED_SET_CPU)` by contrast needs
   uid 0 — `caps_priv_check`, `kern_caps.c` — the prompt's prior-session
   assumption held only for the syscall, which is enough).

## Why the wedge did not close (quantified gap)

- **rdyprocmask governor**: `bsd4_kick_helper` checks and clears the
  target's `bsd4_rdyprocmask` bit (:1601-1609) *before* counting a kick,
  so a single spin-held scan can send at most ~1 IPI per target cpu; the
  target's helper re-arms the bit each cycle. The bulk Xipiq drain on the
  target then resets the per-sender FIFO to ~0 in one interrupt.
- Achieved max *sustained* (1 Hz-sampled) outgoing depth: **20 / 128**.
- Rate: 48 liveloop entries in ~45 min of adversarial pressure, each <1 s.
  A permanent wedge needs either (a) 129 enqueues inside one target
  drain-gap (governor caps this at ~20 on this guest) or (b) two
  anti-parallel liveloops overlapping within their sub-second windows.
  Neither race closed in 6 storm variants (unpinned, pinned victim/sender,
  interactive victims, pulsed fresh waves, queue_checks=5 and =64).
- `queue_checks=64` (root sysctl) does NOT multiply per-target enqueues
  (the governor caps at 1/scan); it only widens the scan.
- **Mode (a) of the finding (same-cpu schedclock recursion) is
  architecturally impossible**: holding a spinlock implies a critical
  section (`spin_lock` -> `crit_enter_raw`, `spinlock2.h`), and timer
  interrupts pend behind `td_critcount` instead of executing —
  `bsd4_schedulerclock`'s own `KKASSERT(gd->gd_spinlocks == 0)`
  (`usched_bsd4.c:943`) documents exactly this invariant. A schedclock can
  never fire under `bsd4_spin`.
- Mode (b) (cross-cpu mutual wedge: livelooping holder + target parked in
  `_spin_lock_contested`, which spins in a crit section without draining —
  `kern_spinlock.c:167+`) remains *theoretically* reachable but is a
  narrow race that our sustained storms never hit.

## Demonstrated impact (what did reproduce)

- Unprivileged, `kern.user_scheduler=bsd4` boot config:
  - kicks counter 4 -> 238,205 (spin-held kick volume),
  - 48 spin-held FIFO-full liveloops,
  - ssh sessions starved for > 60 s continuously (E2, t+44s onward),
    load average peaking at 71, storm waves stretching from 6 s nominal
    to 2-4 min (20-40x),
  - full self-recovery after the storm; watchdog beats never froze
    (>= 5 s) in any run; `debug.df2712.wmarks` = 0 throughout.

## Fix validation (fix.diff — deferred-kick + queue_checks clamp)

`fix.diff` (v2, minimal shape): kicks are RECORDED under the spin
(`bsd4_record_kick` — atomic ops only, priority snapshot) and ISSUED after
`bsd4_spin` is released (`bsd4_issue_kicks`), only in
`bsd4_select_curproc`'s path; the helper-thread loop keeps its original
plain `bsd4_chooseproc_locked` calls; `queue_checks` is clamped to
[1, BSD4_KICK_MAX-2] in the scan loop (also closes the DF-0019 degenerate
`queue_checks<=0` edge).

- Compiles clean under `-Werror` in the guest kernel build
  (`fixbuild3.log`), boots, bsd4 active, rig armed.
- Semantics preserved under identical storms: kicks 5 -> 27,710 during the
  6-pulse battery (stock: +15,612 on the same battery — kick flow intact,
  even richer), storms complete 6/6 cycles.
- Spin-held IPI issuance is eliminated *by construction*: on the fixed
  kernel the only senders that can still liveloop are the unlocked ones
  (generic by-design backpressure). Residual `ipiq_fifofull` growth under
  the same storms dropped from 3.3/cycle (stock) to 2.2/cycle (fix).
- Blocking-wakeup sanity (corrected two-pipe ping-pong, 20,000 round
  trips/run): stock 9/9 at 2.1-8.3 us/rt; fix 9/9 at 1.8-5.2 us/rt.
- Methodological note recorded below: an intermediate "regression" we
  chased (single-pipe ping-pong deadlocks on every kernel, both
  schedulers, and even on the Linux host) turned out to be a protocol bug
  in the *test* (a bidirectional single pipe lets a side consume its own
  echoed bytes). That evidence is void; the two-pipe test is the valid one.

## Experiments log index

| tag | kernel | knobs | storm | outcome |
|---|---|---|---|---|
| smoke | #1 stock+rig | defaults | storm2 60/30 8s | kicks 4->2088, fifofull 0->1, survived |
| e1 | #1 | defaults | storm 300 unpinned 120s | kicks +0 (no pinning -> no kicks), survived |
| e2 | #1 | defaults | storm2 180/60 120s | +126K kicks in <4s, fifofull +30, ssh dead >60s, recovered, survived |
| e3a/e4a | #1 | defaults | storm3/storm4 100-120s | +56K then frozen; fifofull +3; survived |
| e5 | #1 | qchecks=64 | storm4 90s | +70 kicks (warm rdyprocmask); survived |
| e6 | #1 | qchecks=64 | storm5 9 pulse cycles | +9.6K kicks, fifofull +11 (48 total), cycles 20-40x slow, survived |
| stock smoke/pulses | #4 stock+rig | defaults | same shapes | 2->2154 kicks / +15.6K kicks, fifofull +21, survived |
| fixv2 smoke/pulses | #5 fix+rig | defaults | same shapes | 5->1466 / ->27.7K kicks, fifofull 0->13, survived |
| pp3 (VOID) | #2-#5 + host | - | single-pipe ping-pong | deadlocks everywhere incl. Linux host = test protocol bug |
| pp4 | #4 vs #5 | - | two-pipe ping-pong 9x each | both 9/9, ~2 us/rt |

## Conclusion for triage

The finding's severity rests on the permanent wedge, which we could not
demonstrate. What stands: an objectively unsafe primitive (IPI/wakeup under
the global scheduler spinlock, liveloop with interrupts enabled while
holding it — 48 observed events) plus a demonstrated unprivileged
minutes-grade DoS on bsd4-configured systems, both cleanly removed by the
attached minimal deferred-kick fix, which validates without regression.
 Recommend keeping the finding at Medium (availability wedge class closed
as not demonstrated on this hardware; residual risk is the theoretical
mutual-wedge race and the real transient starvation), and taking the fix.

## Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest `make -j6 nativekernel KERNCONF=X86_64_GENERIC` completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.