DF-3906 — xl(4) AB-BA callout deadlock: xl_stop() vs xl_stats_update()

File: sys/dev/netif/xl/if_xl.c (pass 2, GLM 5.3) Family: DF-3715 / 3738 / 3760 / 3776 / 3806 / 3850 / 3903 (10th instance) — this is the 3Com 3c90x xl(4) member; aggravating delta (like dc/sk): xl_stat_timer is a non-MPSAFE callout.

What to run (on the audit QEMU guest, root)

md5 /usr/src/sys/dev/netif/xl/if_xl.c          # 50da613cb89ad7c4e36eca8304bee221 == audited tree
./build.sh                                      # xl_callout_replica.ko (+ _fixed.ko)
kldload /root/kmod/xl_callout_replica.ko
sysctl kern.xlreplica_arm=1                     # arm the xl_stats_update-shaped 1-tick timer
sleep 1
(sysctl kern.xlreplica_stop=1 >/dev/null 2>&1 &)   # xl_stop shape: NEVER RETURNS

Expected

The background sysctl never returns. A second session shows the stopper thread in state D, wchan costp (the synchronous callout_stop() wait, sys/kern/kern_timeout.c:920), and one (softclock N) thread in state B, wchan slize (parked in lwkt_serialize_enter). Serial console (dfbsd-qemu/boot.log):

xlreplica[arm]: xl_stat_timer armed (1 tick), waiting
xlreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1)
xlreplica[stop]: IF THIS IS THE LAST xlreplica LINE callout_stop NEVER RETURNED (AB-BA deadlock)

No further xlreplica lines appear: callout_stop() never returned. The wedge survives a clean shutdown attempt (the parked softclock thread holds the Giant MP lock because xl_stat_timer is a plain, non-MPSAFE callout, if_xl.c:1320); the guest must be force-reset.

The guest has no 3c90x hardware, so the driver itself is exercised by a kernel module (xl_callout_replica.c) that replicates if_xl.c's exact callout/serializer protocol (same callout flavor, same blocking enter, same self re-arm inside the serialized body, same callout_stop under the serializer). The deadlock is in that generic protocol.

Fix-shape validation (fresh guest)

kldload /root/kmod/xl_callout_replica_fixed.ko   # lwkt_serialize_try + 1-tick retry (fix.diff shape)
sysctl kern.xlrepfix_arm=1; sleep 1
(sysctl kern.xlrepfix_stop=1 >/dev/null 2>&1 &)  # holds serializer 3s, then callout_stop

Expected (observed): xlrepfix[stop]: callout_stop RETURNED (fixed shape): fired=312 ran=5 retry=307 pending=0 — under cross-CPU contention the tick retries instead of parking and callout_stop returns.

DF-3906 — VERDICT

Status: mechanism PROVEN on the guest kernel on the FIRST attempt by a deterministic replica of if_xl.c's exact callout/serializer protocol; xl-specific runtime hardware-gated (no 3Com 3c90x NIC on the audit guest) → inconclusive per family convention. reproduced=0 (no xl hardware on this guest), impact on this guest=none. The class of impact, when it fires on xl(4) hardware, is a permanent kernel deadlock (local DoS): wedged interface, wedged stopper thread, softclock blackout on one CPU, and — because xl_stat_timer is a plain non-MPSAFE callout — the Giant MP lock held by the parked softclock thread. The wedge also defeated the guest's clean shutdown (force reset required). This is the 10th member of the DF-3715 family across the DFly NIC drivers.

Root cause (path:line)

1. sys/dev/netif/xl/if_xl.c:1320 — callout_init(&sc->xl_stat_timer): plain (non-MPSAFE) callout → the callback executes under the Giant MP lock (sys/kern/kern_timeout.c:583 get_mplock()).
2. sys/dev/netif/xl/if_xl.c:2372-2380 — xl_stats_update() does a blocking lwkt_serialize_enter(sc->arpcom.ac_if.if_serializer) at :2377 (sys/kern/lwkt_serialize.c:105 → tsleep, wchan slize), then runs xl_stats_update_serialized() (16 stats register reads + window juggling + mii_tick) and self re-arms at :2429 (callout_reset(..., hz, xl_stats_update, sc), guarded only by xl_stats_no_timeout, which is 0 in the normal 1 Hz tick context; armed from xl_init :2918 on every interface init).
3. sys/dev/netif/xl/if_xl.c:3136-3169 — xl_stop() asserts it holds that serializer (:3142) and then calls the synchronous callout_stop(&sc->xl_stat_timer) at :3169. Every xl_stop() caller holds it: xl_detach :1573, xl_ioctl SIOCSIFFLAGS-down :3065, xl_init :2712 (SIOCSIFFLAGS-up at :3062, media change via xl_ifmedia_upd :2955 → xl_init, watchdog :3125 via xl_init), xl_shutdown :3212, xl_suspend :3222.
4. sys/kern/kern_timeout.c:1091 → :859-932 — callout_stop() is synchronous: when the callout is INPROG (:567), the caller ssleep(c, &c->spin, 0, "costp", 0) (:920) until the callback completes.
5. ⇒ An xl_stats_update dequeued and parked in lwkt_serialize_enter() on the serializer held by the xl_stop caller deadlocks against it (AB-BA): both sleep forever; the interface serializer is never released; the affected CPU's softclock thread stops delivering all callouts; and since xl_stat_timer is non-MPSAFE, the parked softclock thread also holds the Giant MP lock, degrading the whole system.

Trigger on xl(4) hardware: any xl_stop() racing the 1 Hz stats tick — ifconfig xlX down/up, media change on a running interface, watchdog reinit, detach/kldunload, suspend/resume, shutdown. The tick fires at 1 Hz and its serialized body is long (16 windowed register reads + MII bit-banging with DELAYs inside mii_tick), so the window per stop is real; each interface stop is one lottery draw.

This is the xl(4) instance of the DF-3715/3738/3760/3776/3806/3850/3903 family (em, emx, alc, jme, msk, dc, sk — 8 drivers, this is the 9th driver / 10th timer: em alone had two callout sites). Aggravating delta (shared with dc DF-3850 and sk DF-3903): the timer is a non-MPSAFE callout, so the parked callback additionally holds the Giant MP lock.

Verification performed (guest, DragonFly 6.5-DEVELOPMENT x86_64)

• md5 /usr/src/sys/dev/netif/xl/if_xl.c = 50da613cb89ad7c4e36eca8304bee221 — guest source is byte-identical to the audited tree.
• xl_callout_replica.c (this pack) — kernel module replicating the xl protocol byte-for-byte in shape: plain callout_init (non-MPSAFE), tick callback doing blocking lwkt_serialize_enter + self re-arm inside the serialized body, stop sysctl emulating xl_stop (serializer held across the synchronous callout_stop).
• Positive test (xl shape): first attempt reproduced — serial console: xlreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1) then the terminal marker line; no further xlreplica output ever appeared → callout_stop never returned.
• Wedged-thread evidence from a second session: stopper D2 costp sysctl kern.xlreplica_stop=1 (pid 1104; costp is kern_timeout.c:920), and (softclock 4) in B4 slize — the parked non-MPSAFE callback holding the Giant MP lock.
• The wedge survived a clean shutdown attempt — vm.sh down fell back to killing QEMU (guest not answering (likely DDB on panic); killing qemu), confirming the MP-lock aggravation.
• Fix-shape validation on a fresh guest with the exact fix.diff protocol (lwkt_serialize_try + 1-tick retry): same-CPU interleave: callout_stop RETURNED (fixed shape): fired=4 ran=4 retry=0 pending=0; cross-CPU contention (stopper busy-holding the serializer 3 s from another session, the interleave that deadlocks the unfixed shape): callout_stop RETURNED: fired=312 ran=5 retry=307 pending=0 — 307 retries, 0 parks, timer cleanly stopped. Deadlock eliminated; guest stayed healthy and unloadable.

Why a replica (and not the driver)

The guest has no 3Com 3c90x PCI NIC; if_xl.ko cannot even probe. The deadlock, however, lives entirely in the generic callout + lwkt_serializer protocol the driver uses — plain callout, blocking serialize_enter in the callback, synchronous callout_stop under the same serializer — all of which the replica exercises on the live kernel. Same methodology as DF-3715/3738/3760/3776/3806/3850/3903; the mechanism has now been reproduced live 8 times with zero false triggers.

Negative results (pass-2 checklist items killed for if_xl.c)

• RX length trust (DF-3604 family): NOT present. xl_rxeof (:1948-2065) reads total_len = rxstat & XL_RXSTAT_LENMASK (13 bits, NIC-DMA'd) at :1965 but immediately bounds it at :1973: total_len > XL_MAX_FRAMELEN (= ETHER_MAX_LEN + ETHER_VLAN_ENCAP_LEN = 1522, if_xlreg.h:89) sets UP_ERROR|OVERSIZE → dropped at :1982. 1522 < the 2046-byte usable posted cluster (xl_newbuf :1883-1890). No underflow path (m_len = total_len ≥ 0).
• MTU: xl_ioctl has no SIOCSIFMTU case → ether_ioctl caps at ETHERMTU (sys/net/if_ethersubr.c:716), so no jumboMTU abuse.
• TX ring winding: clean. xl_txeof :2072 (boomerang: DOWNLIST_PTR walk) and xl_txeof_90xB :2123 (prod/cons walk) free each mbuf exactly once; xl_start_body :2530-2539 only pops the free list after a successful xl_encap; xl_encap error frees the mbuf and leaves the ring untouched.
• Priority queues: this port has none — if_snd is a single ALTQ queue (no xl_tx_pqueue equivalent exists in the file).
• MII/PHY glue: xl_miibus_readreg :528 bzeros the frame first and returns frame.mii_data — no uninitialized memory reaches callers on MII ack failure (worst case a 0 value).
• EEPROM reads: all xl_read_eeprom call sites are compile-time bounded (eaddr[6] w/ cnt=3, devid w/ cnt=1, xcvr[2] w/ cnt=2, xl_caps w/ cnt=1) — no overflow.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.