# DF-3717 — mxge_detach() callout_stop() AB-BA deadlock vs mxge_tick (protocol replica, guest-verified mechanism)

## What this finding is

`sys/dev/netif/mxge/if_mxge.c`:

- `mxge_tick()` (:3712-3750) is a self-re-arming MP callout (`sc->co_hdl`,
  armed at attach with `callout_reset_bycpu` :4445, re-armed at :3747 with
  `callout_reset` from *inside* the held `sc->main_serialize`). Its **first
  act** (:3720) is a **blocking** `lwkt_serialize_enter(&sc->main_serialize)`.
- `mxge_detach()` (:4454-4517) runs `ifnet_serialize_all(ifp)` (:4463 — this
  takes `main_serialize` first, it is `serializes[0]`, :4133) and then calls
  **synchronous** `callout_stop(&sc->co_hdl)` (:4468) *while holding that
  serializer*.  (`callout_terminate` at :4474 is correctly placed *outside*
  the serializers, but the deadlock happens earlier, at :4468.)

DragonFly's `callout_stop()` (sys/kern/kern_timeout.c:1091-1093) calls
`_callout_cancel_or_stop(cc, CALLOUT_STOP, 1)` — `sync=1` — which sleeps on
`"costp"` (kern_timeout.c:920) until an INPROG callback completes.  A
`mxge_tick` callback that was dequeued and has parked in
`lwkt_serialize_enter` (lwkt_serialize.c:84-90 → `atomic_intr_cond_enter` →
`lwkt_serialize_sleep` → `tsleep(..., "slize", ...)`) can never complete
until the serializer is released — and the stopper holds it while it sleeps.
**AB-BA**: permanent wedge of (a) the detaching thread and (b) the per-CPU
softclock thread of the callback's CPU — all callouts on that CPU stop
firing.  Same mechanism class as DF-3715 (em(4)), here in the mxge(4)
detachment path.

## Guest verification (no mxge hardware — protocol replica)

The guest has no Myricom NIC (no 0x14c1:0x0008/0x0009 PCI device;
`if_mxge.ko` present in /boot/kernel but not loaded), so the driver path
itself cannot run.  Per family convention (DF-3713..DF-3715), the *kernel
primitives* were verified on the guest with an out-of-tree module
(`mxge_callout_replica.c`) that replicates the exact mxge protocol:

- `tick_fix`/`tick_stop` = `mxge_tick` shape: first statement blocking
  `lwkt_serialize_enter`, self re-arm via `callout_reset` inside the lock.
- `fix` sysctl = mxge_detach **with the recommended fix**: enter serializer,
  `callout_stop_async`, release, `callout_terminate` (mirrors :4463 +
  fixed :4468 + existing :4474).
- `stop` sysctl = mxge_detach **as shipped**: enter serializer, sync
  `callout_stop` (:4463-:4468).

## Result (decisive run, DragonFly 6.5-DEVELOPMENT x86_64, hz=100)

Fix shape — **no deadlock, clean stop** (serial console):
```
mxge_replica[fix]: tick parked (started=9 ran=8); callout_stop_async + release + callout_terminate
mxge_replica[fix]: returned; pending=0 ran=9 (0 == cleanly stopped, NO deadlock)
```
(sysctl returned FIX_RC=0.)

Shipped shape — **AB-BA deadlock proven** (serial console; last lines ever
printed by the module):
```
mxge_replica: armed STOP-test tick (hz/4)
mxge_replica[stop]: tick BLOCKED on our serializer (started=9 ran=8) - entering callout_stop() :4468
mxge_replica[stop]: IF THIS IS THE LAST mxge_replica LINE, THE mxge_detach AB-BA DEADLOCK IS PROVEN
```
`callout_stop()` never returned.  Live wedge state observed from a second
session (`ps -ax -o pid,stat,wchan,command`):
```
    -1 B2     slize    (softclock 2)
   859 D1     costp    sysctl kern.mxgereplica.stop=1
```
i.e. the softclock thread of CPU 2 is parked **inside
`lwkt_serialize_enter`** (wchan `slize`, the `mxge_tick` half) while the
"detaching" thread sleeps **inside `callout_stop`'s sync wait** (wchan
`costp`, kern_timeout.c:920 — the `mxge_detach` half) holding the very
serializer the callback needs.  Both wedged permanently; CPU 2's callout
delivery dead.  Guest was reset with `vm.sh reset with-src` afterwards.

### Harness notes (honesty section)
- v1 of the replica re-armed a `callout_terminate`d callout and hit an
  unrelated `panic: assertion "cc->flags & CALLOUT_DID_INIT"` — a harness
  bug (terminate is terminal; mxge never re-arms after detach), fixed by
  using two independent callouts.
- v2 polled for the parked callback with `tsleep(timeout=2)` — the poll
  froze because tsleep timeouts are themselves callouts serviced by the
  very softclock thread that parks (observed live: softclock N in `B2
  slize` while the waiter sat in `D2 mxgw`).  Replaced with a bounded
  busy-wait.  Both artifacts are documented in run.log; neither affects
  the final decisive run, whose protocol matches if_mxge.c exactly.

## Build / run
```
build.sh   # tarball -> /root/kmod on guest; make (DragonFly bsd.kmod.mk)
run.sh     # kldload; fix-shape test; then shipped-shape test (wedges guest)
```
Expected: fix-shape returns `pending=0`; shipped-shape prints the two
`[stop]` lines and never returns; `ps` shows `costp` + `slize` wedge.

## Fix
`fix.diff`: replace the synchronous `callout_stop(&sc->co_hdl)` at
if_mxge.c:4468 with `callout_stop_async(&sc->co_hdl)` (the existing
`callout_terminate(&sc->co_hdl)` at :4474 — already outside the serializers
— then performs the sync drain safely).  Fix-shape validated on the guest
(see above).  `vmxge` cannot be diff-tested without hardware
(fix_status=not_testable per family convention).