# DF-3850 run log — dc(4) AB-BA callout deadlock, mechanism replica (guest)

Guest: DragonFly dfbsd 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 x86_64
(X86_64_GENERIC), 6 CPUs. Guest /usr/src verified byte-identical to the
audited tree before the run:

    MD5 (/usr/src/sys/dev/netif/dc/if_dc.c) = f396b494478178deff15f9ba999ef0b9

## Step 1 — load replica, arm the dc_tick-shaped timer, fire dc_stop shape

    # kldload /root/kmod/dc_callout_replica.ko && echo KLD_OK
    KLD_OK
    # sysctl kern.dcreplica_arm=1
    kern.dcreplica_arm: 0 -> 0        (handler ran; callout armed at 1 tick)
    # (sysctl kern.dcreplica_stop=1 >/dev/null 2>&1 &)   <- run in background

Serial console (dfbsd-qemu/boot.log) — decisive marker lines:

    dcreplica: loaded (if_dc.c callout protocol replica)
    dcreplica[arm]: dc_stat_timer armed (1 tick), waiting
    dcreplica[stop]: tick INPROG and PARKED on the serializer (fired=21 ran=20)
    dcreplica[stop]: IF THIS IS THE LAST dcreplica LINE callout_stop NEVER RETURNED (AB-BA deadlock)

fired=21 vs ran=20: the 21st tick was dequeued (INPROG set,
kern_timeout.c:567) and is parked in lwkt_serialize_enter() — it never
completed, exactly like dc_tick parked on ifp->if_serializer while a
dc_stop caller holds it.

## Step 2 — wedged-thread evidence (captured from a second session)

    # ps -o pid,stat,wchan,command -ax | grep dcreplica_stop | grep -v grep
     1530 D1     costp    sysctl kern.dcreplica_stop=1

    # ps -axH -o pid,tdnam,stat,wchan,command | grep -iE "softclock|dcreplica"
        -1 B0     slize    (softclock 0)        <- parked in lwkt_serialize sleep,
                                                   holding the Giant MP lock
                                                   (non-MPSAFE callout,
                                                   kern_timeout.c:583)
        -1 B1     -        (softclock 1)        <- other CPUs still deliver
        -1 B2     -        (softclock 2)
        -1 B3     -        (softclock 3)
        -1 B4     -        (softclock 4)
        -1 B5     -        (softclock 5)
     1530 D1     costp    sysctl kern.dcreplica_stop=1   <- parked inside
                                                   callout_stop() at
                                                   kern_timeout.c:920

Both halves of the cycle are visible:
  * stopper thread: state D1, wchan `costp`  == ssleep() inside the
    synchronous callout_stop() (sys/kern/kern_timeout.c:920)
  * softclock 0: state B, wchan `slize`      == the tick callback blocked in
    lwkt_serialize_enter() (sys/kern/lwkt_serialize.c:105 ->
    atomic_intr_cond_enter(..., lwkt_serialize_sleep)), INPROG set
    (kern_timeout.c:567), Giant MP lock held (kern_timeout.c:583,
    plain callout_init, if_dc.c:1847)

## Step 3 — collateral: the wedge survives a clean shutdown attempt

`vm.sh down` (ssh `shutdown`): guest never reached poweroff — the shutdown
path itself hung behind the parked mplock/serializer state; the qemu
process had to be SIGKILLed and the disk reverted
(`vm.sh reset with-src`). This matches the DF-3715 family blast radius
(permanent kernel degradation; here aggravated by the MP lock because
dc_stat_timer is a non-MPSAFE callout, unlike em/al/jme/msk).

## Step 4 — fix-shape validation (fixed protocol replica, fresh guest)

    # kldload dc_callout_replica_fixed.ko
    # sysctl kern.dcreplica_armfx=1 ; sleep 1
    # time sysctl kern.dcreplica_stopfx=1
    kern.dcreplica_stopfx: 0 -> 0
            3.09 real         0.00 user         3.09 sys
    dmesg:
    dcreplica-fx: loaded (FIXED if_dc.c protocol replica)
    dcreplica-fx[arm]: fixed-shape dc_stat_timer armed
    dcreplica-fx[stop]: FIXED shape: callout_stop RETURNED, pending=0 fired=318 retried=308 ran=10

With the fix shape (lwkt_serialize_try + 1-tick retry re-arm), the tick
never parks while the stopper holds the serializer: 318 fires, 308 cheap
retries, and callout_stop() returns in exactly the 3 s the handler held
the serializer, with callout_pending()==0 — the STOP post-processing
unloaded the pending retry, so the timer is genuinely dead. No deadlock.

(guest reset to with-src snapshot afterwards)