DF-3903 — sk(4) AB-BA callout deadlock: sk_stop() vs sk_tick()/sk_yukon_tick()

File: sys/dev/netif/sk/if_sk.c (pass 2, GLM 5.3) Family: DF-3715 / 3738 / 3760 / 3776 / 3806 / 3850 (7th instance) — this is the SysKonnect sk(4) member; aggravating delta (like dc): the callout is non-MPSAFE.

What to run (on the audit QEMU guest, root)

md5 /usr/src/sys/dev/netif/sk/if_sk.c          # aff2b52bf166e5ebcbe6354b1ce24e6d == audited tree
./build.sh                                      # sk_callout_replica.ko (+ _fixed.ko)
kldload /root/kmod/sk_callout_replica.ko
sysctl kern.skreplica_arm=1                     # arm the sk_yukon_tick-shaped 1-tick timer
sleep 1
(sysctl kern.skreplica_stop=1 >/dev/null 2>&1 &)   # sk_stop shape: NEVER RETURNS

Expected

The background sysctl never returns. A second session shows the stopper thread in state D, wchan costp (the synchronous callout_stop() wait, sys/kern/kern_timeout.c:920), and one (softclock N) thread in state B, wchan slize (parked in lwkt_serialize_enter). Serial console (dfbsd-qemu/boot.log):

skreplica[arm]: sk_tick_timer armed (1 tick), waiting
skreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1)
skreplica[stop]: IF THIS IS THE LAST skreplica LINE callout_stop NEVER RETURNED (AB-BA deadlock)

No further skreplica lines appear: callout_stop() never returned. The wedge survives a clean shutdown attempt (the parked softclock thread holds the Giant MP lock because sk_tick_timer is a plain, non-MPSAFE callout, if_sk.c:1311); the guest must be force-reset.

The guest has no SysKonnect hardware, so the driver itself is exercised by a kernel module (sk_callout_replica.c) that replicates if_sk.c's exact callout/serializer protocol (same callout flavor, same blocking enter, same unconditional self re-arm, same callout_stop under the serializer). The deadlock is in that generic protocol, which the replica exercises byte-for-byte on the live kernel.

Fix-shape validation (fresh guest)

kldload /root/kmod/sk_callout_replica_fixed.ko   # lwkt_serialize_try + 1-tick retry (fix.diff shape)
sysctl kern.skrepfix_arm=1; sleep 1
(sysctl kern.skrepfix_stop=1 >/dev/null 2>&1 &)  # holds serializer 3s, then callout_stop

Expected (observed): skrepfix[stop]: callout_stop RETURNED (fixed shape): fired=312 ran=5 retry=307 pending=0 — under cross-CPU contention the tick retries instead of parking and callout_stop returns.

DF-3903 — VERDICT

Status: mechanism PROVEN on the guest kernel on the FIRST attempt by a deterministic replica of if_sk.c's exact callout/serializer protocol; sk-specific runtime hardware-gated (no SysKonnect NIC on the audit guest) → inconclusive per family convention. reproduced=0 (no sk hardware on this guest), impact on this guest=none. The class of impact, when it fires on sk(4) hardware, is a permanent kernel deadlock (local DoS): wedged interface, wedged stopper thread, softclock blackout on one CPU, and — because sk_tick_timer is a plain non-MPSAFE callout — the Giant MP lock held by the parked softclock thread. The wedge also defeated the guest's clean shutdown (force reset required).

Root cause (path:line)

1. sys/dev/netif/sk/if_sk.c:1311 — callout_init(&sc_if->sk_tick_timer), per port: plain (non-MPSAFE) callout → the callback executes under the Giant MP lock (sys/kern/kern_timeout.c:583 get_mplock()).
2. sys/dev/netif/sk/if_sk.c:2081-2091 — sk_yukon_tick() does a blocking lwkt_serialize_enter(ifp->if_serializer) at :2087 (sys/kern/lwkt_serialize.c:105 → atomic_intr_cond_enter(..., lwkt_serialize_sleep) → tsleep, wchan slize), then self re-arms at :2089 (UNCONDITIONALLY, every 1 Hz from sk_init :2766) and exits at :2090. Its twin sk_tick() (:2032) does the same blocking enter at :2041 with a conditional self re-arm at :2067 (link-down GPIO poll loop :2061-2070), reached from sk_init-time XMAC interrupts (sk_intr_xmac :2159) and the BCom link-interrupt path (sk_intr_bcom :2138).
3. ifp->if_serializer is the ONE card-global sc->sk_serializer shared by both ports (ether_ifattach(ifp, ..., &sc->sk_serializer) at :1281; bus_setup_intr(..., &sc->sk_serializer) at :1514).
4. sys/dev/netif/sk/if_sk.c:2770-2782 — sk_stop() asserts it holds that serializer (:2778) and then calls the synchronous callout_stop(&sc_if->sk_tick_timer) at :2782. Every sk_stop() caller holds it: sk_ioctl SIOCSIFFLAGS-down :992, sk_init :2606 (SIOCSIFMTU :977, SIOCSIFFLAGS-up :989, sk_ifmedia_upd :937, watchdog sk_watchdog :1758), sk_detach :1540.
5. sys/kern/kern_timeout.c:1091 → :859-932 — callout_stop() is synchronous: when the callout is INPROG (:567), the caller ssleep(c, &c->spin, 0, "costp", 0) (:920) until the callback completes.
6. ⇒ A sk_yukon_tick/sk_tick dequeued and parked in lwkt_serialize_enter() on the serializer held by the sk_stop caller deadlocks against it (AB-BA): both sleep forever; the shared card serializer is never released (both ports of a dual-port card are dead); the affected CPU's softclock thread stops delivering all callouts; and since sk_tick_timer is non-MPSAFE, the parked softclock thread also holds the Giant MP lock, degrading the whole system.

Trigger on sk(4) hardware: any sk_stop() racing the 1 Hz tick — ifconfig skX down/up, MTU change on a running interface (SIOCSIFMTU → sk_init → sk_stop), media change, watchdog reinit, detach/kldunload, suspend/resume, shutdown. The tick fires at 1 Hz and the stop path (BMU stop with SK_TIMEOUT×DELAY(1) poll loops, :2792-2810) spends real time under the serializer, so the window per stop is real; each interface stop is one lottery draw.

This is the sk(4) instance of the DF-3715/3738/3760/3776/3806/3850 family (em, emx, alc, jme, msk, dc). Aggravating delta (shared with dc(4) DF-3850, vs the em/emx/alc/jme/msk members which use callout_init_mp): the timer is a non-MPSAFE callout, so the parked callback additionally holds the Giant MP lock.

Verification performed (guest, DragonFly 6.5-DEVELOPMENT x86_64)

• md5 /usr/src/sys/dev/netif/sk/if_sk.c = aff2b52bf166e5ebcbe6354b1ce24e6d — guest source is byte-identical to the audited tree.
• sk_callout_replica.c (this pack) — kernel module replicating the sk protocol byte-for-byte in shape: plain callout_init (non-MPSAFE), tick callback doing blocking lwkt_serialize_enter + unconditional self re-arm at 1 Hz, stop sysctl emulating sk_stop (serializer held across the synchronous callout_stop).
• Positive test (sk shape): first attempt reproduced — serial console: skreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1) then the terminal marker line; no further skreplica output ever appeared → callout_stop never returned.
• Wedged-thread evidence from a second session: stopper D0 costp sysctl kern.skreplica_stop=1 (pid 2051; costp is kern_timeout.c:920), and (softclock 1) in B1 slize — the parked non-MPSAFE callback holding the Giant MP lock.
• The wedge survived a clean shutdown attempt — vm.sh down fell back to killing QEMU (guest not answering (likely DDB on panic); killing qemu), confirming the MP-lock aggravation.
• Fix-shape validation on a fresh guest with the exact fix.diff protocol (lwkt_serialize_try + 1-tick retry): same-CPU interleave: callout_stop RETURNED (fixed shape): fired=4 ran=4 retry=0 pending=0; cross-CPU contention (stopper busy-holding the serializer 3 s from another session, the interleave that deadlocks the unfixed shape): callout_stop RETURNED: fired=312 ran=5 retry=307 pending=0 — 307 retries, 0 parks, timer cleanly stopped. Deadlock eliminated; guest stayed healthy and unloadable.

Why a replica (and not the driver)

The guest has no SysKonnect SK-98xx/Yukon PCI NIC; if_sk.ko cannot even probe. The deadlock, however, lives entirely in the generic callout + lwkt_serializer protocol the driver uses — plain callout, blocking serialize_enter in the callback, synchronous callout_stop under the same serializer — all of which the replica exercises on the live kernel. This is the same methodology already used to prove DF-3715/3738/3760/3776/3806/ 3850; the mechanism has now been reproduced live 8 times (em, emx, alc, jme, msk, dc, sk, xl) with zero false triggers.

Negative results (pass-2 checklist items killed for if_sk.c)

• RX length trust (DF-3604 family): NOT present. sk_rxeof (:1803-1888) bounds the NIC-DMA'd 16-bit SK_RXBYTES length at :1859: total_len < SK_MIN_FRAMELEN || total_len > max_frmlen rejects, where max_frmlen is ETHER_MAX_LEN (1518 < the 2046-byte usable posted cluster, if_sk.c:821-832) or, in jumbo mode, SK_JUMBO_FRAMELEN (9018) which is exactly the usable size of the posted jumbo buffer (SK_JLEN 9020 = SK_JUMBO_FRAMELEN + ETHER_ALIGN, if_skreg.h:1349-1355, minus the m_adj(ETHER_ALIGN) at :796).
• TX cleanup walk: clean. sk_txeof (:1986-2029) walks cons→prod with an OWN-bit break, frees/unloads exactly once, decrements the same counter sk_encap increments by nsegs; SK_IS_OACTIVE (if_skvar.h:211) keeps maxsegs ≥ SK_NDESC_SPARE so the KASSERT at :1617 cannot fire.
• Jumbo MTU: SIOCSIFMTU caps at SK_JUMBO_MTU (9000, :972); the jumbo pool math (SK_JSLOTS 384 ≥ ring 256) is sound.
• Known pass-1 findings DF-1276/DF-1277 (DMA-map cleanup index; partial ring on non-ENOBUFS) excluded per instructions.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.