DF-4316 — lge_stop()/lge_tick() callout AB-BA deadlock (DF-3715 family, lge member)

What: lge_tick() blocks in lwkt_serialize_enter(ifp->if_serializer) (if_lge.c:984) and re-arms itself inside that serializer (:1017). Every in-tree caller of lge_stop() — which does callout_stop(&sc->lge_stat_timer) (:1442) — holds that same serializer (lge_detach :576-580, lge_init :1211 via SIOCSIFFLAGS :1387 / lge_rxeoc :933 / lge_watchdog :1425, lge_intr :1029, lge_ioctl :1391). DragonFly's callout_stop() is synchronous (kern_timeout.c:1091 → sync=1 → ssleep "costp" :920): it sleeps until an INPROG callback returns. A lge_tick dequeued while a stop-path thread holds the serializer therefore parks inside lwkt_serialize_enter forever, and the stopper sleeps in costp forever holding the serializer — permanent AB-BA deadlock. Collateral: the interface serializer is never released (every later lge path blocks) and the parked CPU's softclock thread stops delivering all callouts on that CPU.

This is the same protocol defect proven for em(4) in DF-3715; this pack proves it deterministically for lge's exact callout shape.

Build / run / expected

Everything runs on the QEMU guest (DragonFly 6.5-DEVELOPMENT x86_64, 6 vCPU):

# on host
dfbsd-qemu/vm.sh up                      # if down
# copy lge_callout_replica.c + Makefile to guest:/root/lgerep
dfbsd-qemu/vm.sh run_root 'cd /root/lgerep && make'          # see build.log

# negative control (recommended-fix shape: try-enter + retry):
dfbsd-qemu/vm.sh run_root 'kldload /root/lgerep/lge_callout_replica.ko; \
    sysctl kern.lgereplica.arm=1; sleep 2; sysctl kern.lgereplica.stop_fix=1'
# EXPECT (dmesg): "lgereplica[fix]: RESULT pending=0 ... does NOT deadlock"
#                and the sysctl command returns (exit 0).

# positive test (exact lge_tick / lge_stop shape):
dfbsd-qemu/vm.sh run_root '(sysctl kern.lgereplica.stop_lge=1 &); sleep 25; \
    ps -axo pid,stat,wchan,command | grep stop_lge; dmesg | tail -8'
# EXPECT (dmesg): "handler PARKED on our serializer ... entering callout_stop()"
#                "IF THIS IS THE LAST lgereplica LINE, ..." -- then silence
#      (ps):      "<pid> D4 costp sysctl kern.lgereplica.stop_lge=1"
# The sysctl never returns; the session that ran it wedges; a second ssh
# session still works (6 vCPU guest) and shows the D4/costp thread.
# After the test the guest is deliberately wedged -> vm.sh reset with-src.

On real lge hardware the trigger is: ifconfig lge0 down (or up → lge_init→lge_stop), kldunload if_lge, watchdog reinit, or RXCMDFIFO-empty reinit, with the 1 Hz stat tick firing inside the stop-path's serialized section. The guest has no LXT1001 NIC (ifconfig -l → vtnet0 lo0), so the in-driver instance cannot be fired there — hence the kernel-module replica of lge's exact callout protocol.

Files

• lge_callout_replica.c — KLD replica (lge_tick shape + fix-shape control)
• Makefile, build.sh, run.sh
• build.log — full module build (BUILD_RC=0, -Werror)
• run.log — control + deadlock sessions + post-wedge captures
• env.txt — guest uname / NICs / compiler
• fix.diff — lge_tick try-enter + 1-tick retry (never blocks)
• VERDICT.md, manifest.json, verdict.json

DF-4316 — VERDICT

Status: mechanism PROVEN on the guest kernel via a deterministic replica of if_lge.c's exact callout/serializer protocol; the lge-specific runtime is hardware-gated (no Level-One LXT1001 NIC on the guest) → inconclusive per DF-3715 family convention, reproduced=0, impact=none-on-this-guest. The class of impact when it fires on lge hardware is a permanent kernel deadlock (local DoS): admin thread D-state forever, interface serializer held forever, one CPU's softclock thread wedged (all callouts on that CPU stop).

Root cause (path:line)

1. sys/dev/netif/lge/if_lge.c:435 — callout_init(&sc->lge_stat_timer); armed at :1319 (lge_init) with period hz, re-armed at :1017 (lge_tick_serialized) — 1 Hz stat tick.
2. sys/dev/netif/lge/if_lge.c:984 — lge_tick() does a blocking lwkt_serialize_enter(ifp->if_serializer); the work body and the self re-arm (:1017) run inside the serializer; exit at :986.
3. sys/dev/netif/lge/if_lge.c:1442 — lge_stop() calls callout_stop(&sc->lge_stat_timer) while holding that serializer. Every in-tree lge_stop() caller holds it: - lge_detach :576-580 (explicit serialize_enter around lge_reset/lge_stop) - lge_init :1211 — invoked from SIOCSIFFLAGS :1387, lge_rxeoc :933 (from lge_intr), lge_watchdog :1425, if_up - lge_intr :1029 (!(IFF_UP) → lge_stop; intr runs under ifp->if_serializer, handed to bus_setup_intr at :553-555) - lge_ioctl :1391 (SIOCSIFFLAGS down transition) (Only lge_shutdown :1486 calls lge_stop without the serializer — no AB-BA there, stopper holds nothing.) Additionally lge_intr :1055 (PHY_INTR) calls callout_stop() from interrupt-thread context; if the callback is INPROG there, the sync wait (ssleep) happens from an ithread — wedging the driver's interrupt thread.
4. sys/kern/kern_timeout.c:1091-1094 — callout_stop() → _callout_cancel_or_stop(..., CALLOUT_STOP, sync=1); when the callout is INPROG the caller ssleep(c, &c->spin, 0, "costp", 0) at :920 until the callback completes.
5. ⇒ If lge_tick is dequeued (INPROG) while a stop-path thread holds the serializer: the callback parks spinning in lwkt_serialize_enter waiting for the serializer; the stopper sleeps on "costp" waiting for the callback; neither can proceed. The serializer is held forever (all later lge paths block), and the softclock thread of the CPU that armed the timer stops delivering every callout on that CPU (tsleep timeouts included).

Race window: the 1 Hz tick must be dequeued while a stop-path caller is between serialize_enter and callout_stop — once parked it stays parked, so the effective window is the whole serialized stop-path preamble up to :1442 (µs-ms per stop, one die-roll per second of uptime per stop invocation). Triggers on hardware: ifconfig lge0 down/up, MTU change, kldunload if_lge/detach, watchdog reinit, RXCMDFIFO-empty reinit, PHY interrupts. Repeatable admin/hardware events × 1 Hz tick ⇒ eventual occurrence on long-lived boxes is plausible, exactly as argued for DF-3715.

Verification performed (guest, DragonFly 6.5-DEVELOPMENT x86_64, 6 vCPU)

• Built lge_callout_replica.ko on the guest against /usr/src/sys (build.log, BUILD_RC=0, -Werror clean). The module replicates lge's exact protocol: one callout, period hz (=:1319/:1017), callback = blocking lwkt_serialize_enter → work + self re-arm inside → exit (=:978-987); stopper sysctl = serializer acquire (as every lge_stop caller) → callout_stop() (=:1442) → release.
• Negative control (kern.lgereplica.stop_fix): the recommended-fix shape (lwkt_serialize_try + 1-tick retry) — callout_stop() returned while the stopper held the serializer, pending=0; sysctl exited 0. This also live-validates the fix mechanism on this kernel.
• Positive test (kern.lgereplica.stop_lge): after arming, the stopper printed handler PARKED on our serializer (fired=14 ran=13) - entering callout_stop() then IF THIS IS THE LAST lgereplica LINE, THE LGE_STOP AB-BA DEADLOCK IS PROVEN — and nothing more. The sysctl command never returned; its whole ssh session wedged; a second ssh session showed 1223 D4 costp sysctl kern.lgereplica.stop_lge=1 (state D4, wchan costp = kern_timeout.c:920), captured in run.log. Deterministic, not probabilistic: the harness waits for the park before calling callout_stop.
• lge hardware absent (ifconfig -l → vtnet0 lo0) — the if_lge.c instance of the race itself cannot be fired on this guest.
• Guest reset (vm.sh reset with-src) after the wedge; guest_dirty=0.

Fix

fix.diff: make lge_tick() never block — lwkt_serialize_try; on failure re-arm in 1 tick and return (same shape em(4)'s txgc timer uses, validated live by the negative control). With that, callout_stop()'s sync wait always completes and the AB-BA edge is gone. Full end-to-end fix validation would require lge hardware (absent) → fix_status=not_testable; the replica control demonstrates the fixed protocol does not deadlock under the identical stop-while-held-serializer scenario.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.