rl(4): rapid IFF_DOWN/UP churn leaves TX completions permanently missed - recurring watchdog timeouts and multi-second TX stalls (reproduced on emulated rtl8139)
Summary
300 rapid ifconfig down/up cycles leave the driver/chip pair in a state where queued TX descriptors never complete: 11 watchdog timeouts over two churn sessions, DNS queries needing 4s retransmits, one ~30s hard TX stall; recovery only via the watchdog-driven rl_init. if_timer is armed solely by rl_start, so each watchdog proves a missed TXSTAT completion. Likely root: rl_list_tx_init zeroes TXADDR0-3 but never clears TXSTAT0-3 and rl_init never issues rl_reset, so stale completion bits survive every re-init. REPRODUCED 2/2 sessions on the QEMU-emulated rtl8139 (temp launcher); guest stayed up (no corruption). Driver-vs-model attribution honest 'likely'. Clear-TXSTAT fix.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4236 Β· 9 files| File | Type | Description | Size | |
|---|---|---|---|---|
| rl_live_test.sh | β | 1.5 KB | view raw | |
| rl_repro.sh | β | 790 B | view raw | |
| run.log | β | 3.2 KB | view raw | |
| guest_env.txt | β | 1.1 KB | view raw | |
| env.txt | β | 533 B | view raw | |
| fix.diff | β | 489 B | view raw | |
| VERDICT.md | β | 4.5 KB | β raw | |
| README.md | β | 2.0 KB | β raw | |
| verdict.json | β | 3.3 KB | view raw |
DF-4236 β rl(4): rapid IFF_DOWN/UP churn leaves TX completions permanently missed β recurring watchdog timeouts / multi-second TX stalls (reproduced on emulated rtl8139)
File: sys/dev/netif/rl/if_rl.c
Severity: Low (availability; trigger requires ifconfig β i.e. root β but the same state machine also runs from rl_watchdog()/rl_intr())
Status: reproduced live on the QEMU-emulated rtl8139 (impact=dos, confidence=likely β reproduced 2/2 churn sessions; root-cause attribution between driver and QEMU's device model is not fully pinned, see VERDICT.md)
Reproduce
# host: boot guest with the emulated 8139 (temp launcher, vm.sh untouched): dfbsd-qemu/vm.sh down && dfbsd-qemu/vm_rl.sh # guest (root): ifconfig rl0 10.0.2.16 netmask 255.255.255.0 up route add -host 10.0.2.3 -iface rl0 # slirp virtual DNS as traffic peer nslookup -timeout=3 localhost 10.0.2.3 # works (baseline) j=0; while [ $j -lt 300 ]; do ifconfig rl0 down; ifconfig rl0 up; j=$((j+1)); done dmesg | tail # -> "rl0: watchdog timeout" nslookup -timeout=3 localhost 10.0.2.3 # stalled/retrying (4 s apart) netstat -I rl0 -b # Oerrs climbing
Expected: baseline DNS works; after the churn, rl0: watchdog timeout
appears, Oerrs climbs (6 β 11 across two sessions here), queries need
multiple 4-second retransmits; one session showed a ~30 s hard TX stall
that self-healed through the watchdog-driven rl_init(). No panic, no
memory corruption.
What it means
if_timer is only armed by rl_start() when frames are handed to the
chip (if_rl.c:1375), so each watchdog is proof that a queued descriptor
never completed. After churn the chip/driver pair intermittently stops
completing TX descriptors; every 5 s rl_watchdog() (if_rl.c:1562-1573)
does rl_txeof + rl_rxeof + rl_init and recovers β at the cost of dropping
everything queued. See VERDICT.md for the attribution analysis (driver-side
rl_list_tx_init() leaves stale TxStatus completion bits; QEMU's model
vs real 8139 behaviour).
DF-4236 VERDICT β churn-induced missed TX completions / watchdog storms on rl(4)
Verdict: reproduced (dos, Low). Observed 2/2 churn sessions on the
QEMU-emulated rtl8139: recurring rl0: watchdog timeout, Oerrs climbing
(6β11), queries needing 4 s retransmits, one ~30 s hard TX stall that
self-healed via the watchdog-driven rl_init(). No panic, no memory
corruption, guest stayed up. Root-cause attribution driver-vs-QEMU-model is
likely but not fully pinned β the honest statement is "the shipped driver
+ this (widely used) device model enter a degraded state together"; the
same churn against real silicon is untested.
Observation
- Baseline (no churn): DNS over rl0 works, 0 oerrors, no watchdogs (~1250 packets over the session, incl. 1400-byte EDNS frames).
- 300 rapid
ifconfig rl0 down; upcycles βrl0: watchdog timeout(5 messages, session 1; 5 more, session 2 β 11 total),Oerrs0β11, retransmit gaps of exactly 4 s (nslookup timeout), one ~30 s complete TX stall (session 1 final probe timed out 3Γ) that later self-healed without any action beyond furtherrl_init()s. if_timeris armed only byrl_start()(if_rl.c:1375) and rearmed byrl_txeof()while TX is pending (if_rl.c:1151-1154); therefore every watchdog observation proves frames were dequeued into the chip and their completion was never seen byrl_txeof()'sRL_LAST_TXSTATtest (if_rl.c:1122-1125) within 5 seconds.- Recovery always occurred via
rl_watchdog()βrl_init()(if_rl.c:1570-1572), i.e. the failure state is chip/driver state that a full register reprogram clears (noterl_init()does not issuerl_reset()).
Attribution analysis (why "likely", not "certain")
Driver-side facts (sys/dev/netif/rl/if_rl.c):
- rl_stop() (:1591-1592) stops the engine with RL_COMMAND=0, IMR=0
and rl_list_tx_init() (:1418 β :935-949) resets cur_tx/last_tx
and zeroes RL_TXADDR0-3 β but RL_TXSTAT0-3 are never cleared.
Stale completion bits (TOK/OWN-threshold from the previous session, plus
stale byte counts) survive every re-init.
- rl_init() does not call rl_reset(), so any internal descriptor/timer
state inside the chip that keys off TxStatus is carried across the churn.
- rl_txeof() gates on (txstat & (TX_OK|UNDERRUN|TXABRT)) == 0 β a
descriptor the model believes "not yet complete" stalls the slot until
the watchdog re-inits everything.
- With 4 slots and RL_INC mod-4 bookkeeping, a single missed completion
wedges cur_tx == last_tx rotation for that slot pairing and every
subsequent packet waits on it.
Emulator-side facts: QEMU's rtl8139 (non-C+ mode) transmits on the
TxStatus write and updates status registers from its model state; how it
reacts to TxAddr=0 with stale TxStatus bits (exactly what
rl_list_tx_init produces mid-churn) is the kind of corner where a model
and a driver can desync. Real 8139 silicon is specified to trigger on the
driver's descriptor write, which makes the stale-status window mostly
benign on hardware β but that is exactly the untested half.
Given the trigger needs ifconfig (root), the audit rubric puts this at
Low. It is filed because (a) it is a reproduced availability defect on
the audit's own reference emulation, (b) the same re-init state machine
runs unprivileged-triggers-adjacent via rl_watchdog() and
rl_intr(SYSTEM_ERR), and (c) it gives maintainers a concrete, cheap
experiment (clear TxStatus in rl_list_tx_init, or add rl_reset() on
re-init) β see fix.diff candidate.
Fix validation
fix_status = not_testable: the candidate fix (clear RL_TXSTAT0-3 in
rl_list_tx_init) is a one-line, build-safe change, but validating it
requires a kernel rebuild + re-churn cycle whose outcome cannot
distinguish "fix works" from "QEMU model happened to behave" without
hardware cross-checks; the audit contract's kernel-rebuild budget is
reserved for memory-corruption findings. Left as a recommended fix.
Artifacts
run.log (both sessions, netstat/dmesg/tcpdump excerpts), rl_live_test.sh, rl_repro.sh (in DF-4235/, shared session), guest_env.txt (attach proof, final counters), dfbsd-qemu/vm_rl.sh (temp launcher).
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testableCandidate one-line fix authored (fix.diff) but not kernel-rebuild-validated: with root cause not fully pinned to the driver, a patched-guest run cannot distinguish 'fix works' from 'model artifact'; rebuild budget reserved for memory-corruption findings per contract.
fix.diff; baseline reproduction in run.log
Confirmed kernel references
Detail
Exploit chain
root (or any path driving repeated IFF_DOWN/UP, e.g. watchdog cascade) -> churn -> missed TX completions -> 5s watchdog stalls recurring indefinitely -> interface availability degraded until quiescence.
Evidence (decisive lines)
run.log (both sessions: counters, dmesg watchdog lines, tcpdump 4s retransmit gaps, /var/log/messages), rl_live_test.sh + rl_repro.sh (exact reproducers), guest_env.txt (attach + final counters), env.txt (emulator/device IDs), dfbsd-qemu/vm_rl.sh (temp launcher)
PoC changes
n/a (first observation; harness authored this run)
Verified recommended fix
Clear RL_TXSTAT0-3 in rl_list_tx_init() so re-init cannot inherit stale descriptor state (candidate, see fix.diff); validate on real silicon.
Verdict
Reproduced 2/2 on the QEMU-emulated rtl8139: 300 rapid ifconfig rl0 down/up cycles leave rl(4) in a state where TX completions are intermittently never observed -- recurring 'rl0: watchdog timeout' (11 total), Oerrs climbing 0->11, DNS queries needing 4s retransmits, and one ~30s hard TX stall that self-healed only via the watchdog-driven rl_init(). No panic, no corruption, guest stayed up. Every watchdog is proof of frames queued by rl_start (if_rl.c:1375) whose RL_LAST_TXSTAT completion test (if_rl.c:1122-1125) never fired for 5s. Attribution driver-vs-QEMU-model is likely-but-not-pinned: source-side, rl_list_tx_init (:935-949) zeroes RL_TXADDR0-3 but NEVER clears RL_TXSTAT0-3, and rl_init does not rl_reset, so stale completion bits survive every re-init -- exactly the corner where the emulator's descriptor state and the driver desync; real silicon may be benign. Low severity: trigger is ifconfig (root), though the identical re-init state machine also runs unprivileged-adjacent via rl_watchdog/rl_intr.
No comments yet.