# DF-4109 run log — tl(4) callout AB-BA replica (guest, DragonFly 6.5-DEVELOPMENT x86_64)

## Presence probe (hardware gate)
```
# uname -a
DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul  2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC  x86_64
# ifconfig -l
vtnet0 lo0
```
No ThunderLAN (tl) device on the guest -> the driver's exact protocol is
replicated by /root/kmod/tlnge_replica.ko (plain callout_init as
if_tl.c:1197; callback blocks in lwkt_serialize_enter as if_tl.c:1671;
re-arms inside the serializer as if_tl.c:1722 with period hz; stopper
holds the serializer and calls callout_stop as tl_stop/if_tl.c:2121).

## Build
```
# cd /root/kmod && make
... cc -O2 -Werror ... -c tlnge_replica.c
cc -Wl,--build-id=sha1 -nostdlib -Wl,--hash-style=sysv -r -Wl,-d -o tlnge_replica.ko tlnge_replica.o
BUILD_RC=0
-rw-r--r-- 1 root wheel 10448 Sep 18 00:06 tlnge_replica.ko
```

## Run 1 — load, arm, negative control (ordering mistake kept for honesty)
```
# kldload /root/kmod/tlnge_replica.ko && echo KLD_OK; sysctl kern.tlreplica.arm=1
KLD_OK
kern.tlreplica.arm: 0 -> 0
replica[tl]: armed (period hz=100)
# sysctl kern.tlreplica.stop_plain=1
kern.tlreplica.stop_plain: 0 -> 0
replica[tl]: stop_plain entering callout_stop()
replica[tl]: stop_plain callout_stop RETURNED pending=0 (negative control ok)
# ps -o pid,stat,wchan,command -ax | grep stop_blk | grep -v grep
  1583 D0     blkw     sysctl kern.tlreplica.stop_blk=1
# dmesg | grep replica[tl] | tail
replica[tl]: handler never fired/parked
```
(stop_plain had cancelled the timer, so the stop_blk of run 1 only hit my
own wait loop; thread state "blkw" = my spin, not the kernel wedge. This
is why run 2 re-arms first. The negative control itself passed:
callout_stop without the serializer returns, pending=0.)

## Run 2 — clean AB-BA (fresh arm, then stop under serializer)
```
# sysctl kern.tlreplica.arm=1; sleep 3
kern.tlreplica.arm: 0 -> 0
# (sysctl kern.tlreplica.stop_blk=1 >/dev/null 2>&1 &); sleep 6
# dmesg | grep 'replica\[tl\]' | tail -4
replica[tl]: handler never fired/parked
replica[tl]: armed (period hz=100)
replica[tl]: handler PARKED on our serializer (fired=4 ran=3) - entering callout_stop()
replica[tl]: IF THIS IS THE LAST replica LINE, AB-BA DEADLOCK IS PROVEN
# ps -o pid,stat,wchan,command -ax | grep stop_blk | grep -v grep
  1603 D1     costp    sysctl kern.tlreplica.stop_blk=1
```
The stopper thread is parked at kern_timeout.c:920 ("costp") holding the
serializer; the softclock thread that dequeued the tl-shaped handler is
parked in lwkt_serialize_enter ("slize", lwkt_serialize.c:234).
callout_stop() never returned.

## Permanence (+10 s and later)
```
# sleep 10; dmesg | grep 'replica\[tl\]' | tail -2
replica[tl]: handler PARKED on our serializer (fired=4 ran=3) - entering callout_stop()
replica[tl]: IF THIS IS THE LAST replica LINE, AB-BA DEADLOCK IS PROVEN
(nothing further; ALIVE probe answered, wedge persists)
# ps ... | grep stop_blk | grep -v grep
  1603 D1     costp    sysctl kern.tlreplica.stop_blk=1
```

## Collateral damage observed
* A later `sysctl kern.tlreplica` read hung its whole ssh session (the
  wedged sysctl writer never releases the sysctl write lock) — the wedge
  poisons the sysctl subsystem, not just the interface.
* `vm.sh reset` (clean shutdown) timed out both times on wedged guests;
  recovery required force-killing QEMU. On real hardware the equivalent
  is a system that can no longer be shut down cleanly.

## Fix-shape control (post-reset, same module, non-wedging)
```
# sysctl kern.tlreplica.stop_fix=1
kern.tlreplica.stop_fix: 0 -> 0
replica[tl]: fix-shape missing (f=1 r=0) - callout_stop under serializer
replica[tl]: fix-shape callout_stop RETURNED (no deadlock)
replica[tl]: fix-shape RESULT pending=0 fired-stalled=YES (f=1->1 r=0)
```
The proposed fix pattern (lwkt_serialize_try + 1-tick retry) is fully
stopped by callout_stop under a held serializer on this exact kernel:
no deadlock, pending=0, firing stalled.