DF-2712 — evidence pack

Finding: sys/kern/usched_bsd4.c:1590-1615 — bsd4_kick_helper() issues lwkt_send_ipiq()/wakeup() while the caller still holds the global bsd4_spin, so a FIFO-full lwkt_send_ipiq3() liveloop (which enables interrupts and spins, lwkt_ipiq.c:240-277) executes with the scheduler spinlock held — claimed to enable a same-cpu schedclock recursion and a cross-cpu mutual whole-system wedge, unprivileged, when usched_bsd4 is the active user scheduler.

What is in here

Reproduce (summary)

1. ./build.sh — boots guest into a rig kernel with kern.user_scheduler="bsd4".
2. Confirm: sysctl debug.df2712.armed = 1, kern.usched_bsd4.kicks moves under load (bsd4 active; stock boots dfly so the loader switch is required).
3. Smoke (the primitive, 8 s, unprivileged): vm.sh run_root 'su -m maxx -c "/tmp/storm2 5 60 30 8"' then sysctl lwkt.ipiq_fifofull — rose from 0 to 1+ on the stock-bsd4 kernel (48 total across all our runs): spin-held FIFO-full liveloops entered. Each drained < 1 s (no tgt not draining console lines).
4. Wedge attempt (did NOT wedge): su -m maxx -c "/tmp/storm5 5 180 12 3 15".
5. Full battery + fix A/B: see VERDICT.md.

Expected output (what we actually observed)

• kern.usched_bsd4.kicks: 4 -> 238,005 under unprivileged storms.
• lwkt.ipiq_fifofull: 0 -> 48 (spin-held liveloop entries, all transient).
• max outgoing FIFO depth seen by the 1 Hz watchdog sampler: 20 of the 128 (MAXCPUFIFO/2, lwkt_ipiq.c:225) entries needed to enter the liveloop (253 applies to the nested case only).
• ssh/serial starvation > 60 s, load average 30-71, storm cycles stretched from 6 s nominal to 2-4 min — severe recoverable DoS.
• NO permanent wedge: watchdog beats never froze >= 5 s (debug.df2712.wmarks = 0), zero send_ipiq ... tgt not draining lines.

DF-2712 — verification verdict

Bottom line

The claimed whole-system wedge did NOT reproduce. The mechanism it is built on is real and was reproduced line-accurately — 238,205 spin-held bsd4_kick_helper executions and 48 spin-held FIFO-full lwkt_send_ipiq3() liveloops (interrupts enabled, spinning, while the global bsd4_spin is held) — but on this 6-vCPU guest every liveloop drained in under a second, the outgoing FIFO depth never exceeded 20 of the 128 entries needed to enter the liveloop as a sustained state, and the scheduler-spin watchdog never observed a frozen cpu. Alongside the transient liveloops the unprivileged storm produced minutes-scale ssh starvation and load-average 30-71 — a severe recoverable availability degradation, not the claimed permanent hang. An honest inconclusive for the wedge claim, with the gap quantified below.

Line-verified mechanism (all confirmed on the running kernel)

1. bsd4_kick_helper (sys/kern/usched_bsd4.c:1590-1615) executes lwkt_send_ipiq (:1611) / wakeup (:1613) while the caller holds bsd4_spin (its own comment :1586 admits it). Call sites inside bsd4_chooseproc_locked_cache_coherent at :1473/:1512/:1519/:1561, which runs under the spin taken at :533 (bsd4_select_curproc).
2. That path is LIVE under bsd4 by default: usched_bsd4_cache_coherent is set to 1 at registration (usched_bsd4.c:2030) whenever the cpu topology provides nodes (true on this QEMU guest: usched_bsd4.smt: NOT SUPPORTED + cache_coherent: 1).
3. lwkt_send_ipiq3 (sys/kern/lwkt_ipiq.c:181): with gd_processing_ipiq == 0 the liveloop threshold is MAXCPUFIFO/2 = 128 (:225), not 253 (253 is the nested case, :228). On > 128 outstanding it does cpu_enable_intr() (:240) and liveloops (:244-277) until the target drains to <= 64 — still holding bsd4_spin.
4. Reproduced: lwkt.ipiq_fifofull rose 0 -> 48 across the unpriv storms (each entry = one spin-held liveloop). All exited < 1 s: zero send_ipiq %d->%d tgt not draining console lines across every run (that kprintf fires after 1 s stuck, lwkt_ipiq.c:258-275).
5. Unprivileged reachability confirmed: lwp_setaffinity(2) with pid=0 is explicitly allowed for self (kern_usched.c:423 "Always allow change self CPU affinity"; usched_set(USCHED_SET_CPU) by contrast needs uid 0 — caps_priv_check, kern_caps.c — the prompt's prior-session assumption held only for the syscall, which is enough).

Why the wedge did not close (quantified gap)

• rdyprocmask governor: bsd4_kick_helper checks and clears the target's bsd4_rdyprocmask bit (:1601-1609) before counting a kick, so a single spin-held scan can send at most ~1 IPI per target cpu; the target's helper re-arms the bit each cycle. The bulk Xipiq drain on the target then resets the per-sender FIFO to ~0 in one interrupt.
• Achieved max sustained (1 Hz-sampled) outgoing depth: 20 / 128.
• Rate: 48 liveloop entries in ~45 min of adversarial pressure, each <1 s. A permanent wedge needs either (a) 129 enqueues inside one target drain-gap (governor caps this at ~20 on this guest) or (b) two anti-parallel liveloops overlapping within their sub-second windows. Neither race closed in 6 storm variants (unpinned, pinned victim/sender, interactive victims, pulsed fresh waves, queue_checks=5 and =64).
• queue_checks=64 (root sysctl) does NOT multiply per-target enqueues (the governor caps at 1/scan); it only widens the scan.
• Mode (a) of the finding (same-cpu schedclock recursion) is architecturally impossible: holding a spinlock implies a critical section (spin_lock -> crit_enter_raw, spinlock2.h), and timer interrupts pend behind td_critcount instead of executing — bsd4_schedulerclock's own KKASSERT(gd->gd_spinlocks == 0) (usched_bsd4.c:943) documents exactly this invariant. A schedclock can never fire under bsd4_spin.
• Mode (b) (cross-cpu mutual wedge: livelooping holder + target parked in _spin_lock_contested, which spins in a crit section without draining — kern_spinlock.c:167+) remains theoretically reachable but is a narrow race that our sustained storms never hit.

Demonstrated impact (what did reproduce)

• Unprivileged, kern.user_scheduler=bsd4 boot config:
• kicks counter 4 -> 238,205 (spin-held kick volume),
• 48 spin-held FIFO-full liveloops,
• ssh sessions starved for > 60 s continuously (E2, t+44s onward), load average peaking at 71, storm waves stretching from 6 s nominal to 2-4 min (20-40x),
• full self-recovery after the storm; watchdog beats never froze (>= 5 s) in any run; debug.df2712.wmarks = 0 throughout.

Fix validation (fix.diff — deferred-kick + queue_checks clamp)

fix.diff (v2, minimal shape): kicks are RECORDED under the spin (bsd4_record_kick — atomic ops only, priority snapshot) and ISSUED after bsd4_spin is released (bsd4_issue_kicks), only in bsd4_select_curproc's path; the helper-thread loop keeps its original plain bsd4_chooseproc_locked calls; queue_checks is clamped to [1, BSD4_KICK_MAX-2] in the scan loop (also closes the DF-0019 degenerate queue_checks<=0 edge).

• Compiles clean under -Werror in the guest kernel build (fixbuild3.log), boots, bsd4 active, rig armed.
• Semantics preserved under identical storms: kicks 5 -> 27,710 during the 6-pulse battery (stock: +15,612 on the same battery — kick flow intact, even richer), storms complete 6/6 cycles.
• Spin-held IPI issuance is eliminated by construction: on the fixed kernel the only senders that can still liveloop are the unlocked ones (generic by-design backpressure). Residual ipiq_fifofull growth under the same storms dropped from 3.3/cycle (stock) to 2.2/cycle (fix).
• Blocking-wakeup sanity (corrected two-pipe ping-pong, 20,000 round trips/run): stock 9/9 at 2.1-8.3 us/rt; fix 9/9 at 1.8-5.2 us/rt.
• Methodological note recorded below: an intermediate "regression" we chased (single-pipe ping-pong deadlocks on every kernel, both schedulers, and even on the Linux host) turned out to be a protocol bug in the test (a bidirectional single pipe lets a side consume its own echoed bytes). That evidence is void; the two-pipe test is the valid one.

Experiments log index

Conclusion for triage

The finding's severity rests on the permanent wedge, which we could not demonstrate. What stands: an objectively unsafe primitive (IPI/wakeup under the global scheduler spinlock, liveloop with interrupts enabled while holding it — 48 observed events) plus a demonstrated unprivileged minutes-grade DoS on bsd4-configured systems, both cleanly removed by the attached minimal deferred-kick fix, which validates without regression. Recommend keeping the finding at Medium (availability wedge class closed as not demonstrated on this hardware; residual risk is the theoretical mutual-wedge race and the real transient starvation), and taking the fix.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.