DF-5740 / panic.txt
DECISIVE PANIC (race6, unfixed production code, INVARIANTS kernel):
[boot 01:26, module hci_sock_harness, unit attached, injector on cpu3]
[harness] === fire: 99999 injector rounds (8 events each) from cpu 3 ===
panic: receive 1
cpuid = 4
Trace beginning at frame 0xfffff80117cab718
soreceive() at soreceive+0xd1f 0xffffffff806c647f
kern_recvmsg() at kern_recvmsg+0xf7 0xffffffff806cba17
sys_recvfrom() at sys_recvfrom+0xc4 0xffffffff806cbc64
syscall2() at syscall2+0x11e 0xffffffff80bd6a0e
boot() called on cpu#4
Uptime: 24s
[crash dump written to /var/crash/vmcore.0]
MECHANISM (soreceive+0xd1f == uipc_socket.c:1348):
KASSERT(m != NULL || !so->so_rcv.ssb_cc, ("receive 1"));
The tap socket's receive sockbuf is EMPTY (ssb_mb == NULL) but its byte
counter is NON-ZERO: hci_mtap()'s sbappendaddr() (hci_socket.c:1004,
running on CPU3's netisr port inside btintr->hci_output_cmd/hci_intr)
mutated the sockbuf with NO ssb_token/ssb_lock while the user thread on
CPU4 was inside soreceive() on the same sockbuf. The torn
append/consume interleaving left the accounting desynchronized.
CORROBORATING CRASHES/WEDGES (unfixed code, earlier rig iterations with
non-draining taps -- confounded by the unbounded-tap mbuf exhaustion of
DF-5742, noted honestly):
race1/race2 (48/24 taps, no drain): after "Warning: objcache(mbuf
pkthdr) exhausted on cpu0!/cpu3!":
Fatal trap 12: page fault while in kernel mode
cpuid = 3
fault virtual address = 0x8
fault code = supervisor write data, page not present
Stopped at hci_input_event+0x42: movq $0,0x8(%r12)
Trace: hci_input_event() <- inj_thread()
(decoded: IF_ENQUEUE's m->m_nextpkt=NULL with m==NULL after m_gethdr
returned NULL under pool exhaustion; KKASSERT is INVARIANTS-only in the
kmod and _m_copyback2(NULL,..) returns success -- rig artifact enabled
by DF-5742's unbounded tap, not by itself a production path)
race2 aftermath: CPU0 wedged in-kernel, not draining cross-IPIs for 60+
seconds while the panic cpu tried to stop it:
send_ipiq 3->0 tgt not draining (1..69)
race4/race5: objcache(mbuf pkthdr) exhausted on cpu0/cpu1/cpu2/cpu3,
then HARD SYSTEM WEDGE (no trap, no console output, ssh dead).
race5 send-side corruption evidence: non-EWOULDBLOCK send failures
accumulating (96 by iter 875) from the raced so_snd bookkeeping.