β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-4236

rl(4): rapid IFF_DOWN/UP churn leaves TX completions permanently missed - recurring watchdog timeouts and multi-second TX stalls (reproduced on emulated rtl8139)

Summary

300 rapid ifconfig down/up cycles leave the driver/chip pair in a state where queued TX descriptors never complete: 11 watchdog timeouts over two churn sessions, DNS queries needing 4s retransmits, one ~30s hard TX stall; recovery only via the watchdog-driven rl_init. if_timer is armed solely by rl_start, so each watchdog proves a missed TXSTAT completion. Likely root: rl_list_tx_init zeroes TXADDR0-3 but never clears TXSTAT0-3 and rl_init never issues rl_reset, so stale completion bits survive every re-init. REPRODUCED 2/2 sessions on the QEMU-emulated rtl8139 (temp launcher); guest stayed up (no corruption). Driver-vs-model attribution honest 'likely'. Clear-TXSTAT fix.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-4236 Β· 9 files
FileTypeDescriptionSize
rl_live_test.sh β€” 1.5 KB view raw
rl_repro.sh β€” 790 B view raw
run.log β€” 3.2 KB view raw
guest_env.txt β€” 1.1 KB view raw
env.txt β€” 533 B view raw
fix.diff β€” 489 B view raw
VERDICT.md β€” 4.5 KB ↓ raw
README.md β€” 2.0 KB ↓ raw
verdict.json β€” 3.3 KB view raw

DF-4236 β€” rl(4): rapid IFF_DOWN/UP churn leaves TX completions permanently missed β†’ recurring watchdog timeouts / multi-second TX stalls (reproduced on emulated rtl8139)

File: sys/dev/netif/rl/if_rl.c Severity: Low (availability; trigger requires ifconfig β€” i.e. root β€” but the same state machine also runs from rl_watchdog()/rl_intr()) Status: reproduced live on the QEMU-emulated rtl8139 (impact=dos, confidence=likely β€” reproduced 2/2 churn sessions; root-cause attribution between driver and QEMU's device model is not fully pinned, see VERDICT.md)

Reproduce

# host: boot guest with the emulated 8139 (temp launcher, vm.sh untouched):
dfbsd-qemu/vm.sh down && dfbsd-qemu/vm_rl.sh

# guest (root):
ifconfig rl0 10.0.2.16 netmask 255.255.255.0 up
route add -host 10.0.2.3 -iface rl0      # slirp virtual DNS as traffic peer
nslookup -timeout=3 localhost 10.0.2.3   # works (baseline)
j=0; while [ $j -lt 300 ]; do ifconfig rl0 down; ifconfig rl0 up; j=$((j+1)); done
dmesg | tail                              # -> "rl0: watchdog timeout"
nslookup -timeout=3 localhost 10.0.2.3    # stalled/retrying (4 s apart)
netstat -I rl0 -b                         # Oerrs climbing

Expected: baseline DNS works; after the churn, rl0: watchdog timeout appears, Oerrs climbs (6 β†’ 11 across two sessions here), queries need multiple 4-second retransmits; one session showed a ~30 s hard TX stall that self-healed through the watchdog-driven rl_init(). No panic, no memory corruption.

What it means

if_timer is only armed by rl_start() when frames are handed to the chip (if_rl.c:1375), so each watchdog is proof that a queued descriptor never completed. After churn the chip/driver pair intermittently stops completing TX descriptors; every 5 s rl_watchdog() (if_rl.c:1562-1573) does rl_txeof + rl_rxeof + rl_init and recovers β€” at the cost of dropping everything queued. See VERDICT.md for the attribution analysis (driver-side rl_list_tx_init() leaves stale TxStatus completion bits; QEMU's model vs real 8139 behaviour).

VERDICT.md
↓ download raw

DF-4236 VERDICT β€” churn-induced missed TX completions / watchdog storms on rl(4)

Verdict: reproduced (dos, Low). Observed 2/2 churn sessions on the QEMU-emulated rtl8139: recurring rl0: watchdog timeout, Oerrs climbing (6β†’11), queries needing 4 s retransmits, one ~30 s hard TX stall that self-healed via the watchdog-driven rl_init(). No panic, no memory corruption, guest stayed up. Root-cause attribution driver-vs-QEMU-model is likely but not fully pinned β€” the honest statement is "the shipped driver + this (widely used) device model enter a degraded state together"; the same churn against real silicon is untested.

Observation

  • Baseline (no churn): DNS over rl0 works, 0 oerrors, no watchdogs (~1250 packets over the session, incl. 1400-byte EDNS frames).
  • 300 rapid ifconfig rl0 down; up cycles β†’ rl0: watchdog timeout (5 messages, session 1; 5 more, session 2 β€” 11 total), Oerrs 0β†’11, retransmit gaps of exactly 4 s (nslookup timeout), one ~30 s complete TX stall (session 1 final probe timed out 3Γ—) that later self-healed without any action beyond further rl_init()s.
  • if_timer is armed only by rl_start() (if_rl.c:1375) and rearmed by rl_txeof() while TX is pending (if_rl.c:1151-1154); therefore every watchdog observation proves frames were dequeued into the chip and their completion was never seen by rl_txeof()'s RL_LAST_TXSTAT test (if_rl.c:1122-1125) within 5 seconds.
  • Recovery always occurred via rl_watchdog() β†’ rl_init() (if_rl.c:1570-1572), i.e. the failure state is chip/driver state that a full register reprogram clears (note rl_init() does not issue rl_reset()).

Attribution analysis (why "likely", not "certain")

Driver-side facts (sys/dev/netif/rl/if_rl.c): - rl_stop() (:1591-1592) stops the engine with RL_COMMAND=0, IMR=0 and rl_list_tx_init() (:1418 β†’ :935-949) resets cur_tx/last_tx and zeroes RL_TXADDR0-3 β€” but RL_TXSTAT0-3 are never cleared. Stale completion bits (TOK/OWN-threshold from the previous session, plus stale byte counts) survive every re-init. - rl_init() does not call rl_reset(), so any internal descriptor/timer state inside the chip that keys off TxStatus is carried across the churn. - rl_txeof() gates on (txstat & (TX_OK|UNDERRUN|TXABRT)) == 0 β€” a descriptor the model believes "not yet complete" stalls the slot until the watchdog re-inits everything. - With 4 slots and RL_INC mod-4 bookkeeping, a single missed completion wedges cur_tx == last_tx rotation for that slot pairing and every subsequent packet waits on it.

Emulator-side facts: QEMU's rtl8139 (non-C+ mode) transmits on the TxStatus write and updates status registers from its model state; how it reacts to TxAddr=0 with stale TxStatus bits (exactly what rl_list_tx_init produces mid-churn) is the kind of corner where a model and a driver can desync. Real 8139 silicon is specified to trigger on the driver's descriptor write, which makes the stale-status window mostly benign on hardware β€” but that is exactly the untested half.

Given the trigger needs ifconfig (root), the audit rubric puts this at Low. It is filed because (a) it is a reproduced availability defect on the audit's own reference emulation, (b) the same re-init state machine runs unprivileged-triggers-adjacent via rl_watchdog() and rl_intr(SYSTEM_ERR), and (c) it gives maintainers a concrete, cheap experiment (clear TxStatus in rl_list_tx_init, or add rl_reset() on re-init) β€” see fix.diff candidate.

Fix validation

fix_status = not_testable: the candidate fix (clear RL_TXSTAT0-3 in rl_list_tx_init) is a one-line, build-safe change, but validating it requires a kernel rebuild + re-churn cycle whose outcome cannot distinguish "fix works" from "QEMU model happened to behave" without hardware cross-checks; the audit contract's kernel-rebuild budget is reserved for memory-corruption findings. Left as a recommended fix.

Artifacts

run.log (both sessions, netstat/dmesg/tcpdump excerpts), rl_live_test.sh, rl_repro.sh (in DF-4235/, shared session), guest_env.txt (attach proof, final counters), dfbsd-qemu/vm_rl.sh (temp launcher).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

not_testable
baseline reproduced→ patch + rebuild →patched clean

Candidate one-line fix authored (fix.diff) but not kernel-rebuild-validated: with root cause not fully pinned to the driver, a patched-guest run cannot distinguish 'fix works' from 'model artifact'; rebuild budget reserved for memory-corruption findings per contract.

fix.diff; baseline reproduction in run.log
↓ fix.diffper-fix-DF-4236

Confirmed kernel references

Detail

Exploit chain

root (or any path driving repeated IFF_DOWN/UP, e.g. watchdog cascade) -> churn -> missed TX completions -> 5s watchdog stalls recurring indefinitely -> interface availability degraded until quiescence.

Evidence (decisive lines)

run.log (both sessions: counters, dmesg watchdog lines, tcpdump 4s retransmit gaps, /var/log/messages), rl_live_test.sh + rl_repro.sh (exact reproducers), guest_env.txt (attach + final counters), env.txt (emulator/device IDs), dfbsd-qemu/vm_rl.sh (temp launcher)

PoC changes

n/a (first observation; harness authored this run)

Verified recommended fix

Clear RL_TXSTAT0-3 in rl_list_tx_init() so re-init cannot inherit stale descriptor state (candidate, see fix.diff); validate on real silicon.

Verdict

Reproduced 2/2 on the QEMU-emulated rtl8139: 300 rapid ifconfig rl0 down/up cycles leave rl(4) in a state where TX completions are intermittently never observed -- recurring 'rl0: watchdog timeout' (11 total), Oerrs climbing 0->11, DNS queries needing 4s retransmits, and one ~30s hard TX stall that self-healed only via the watchdog-driven rl_init(). No panic, no corruption, guest stayed up. Every watchdog is proof of frames queued by rl_start (if_rl.c:1375) whose RL_LAST_TXSTAT completion test (if_rl.c:1122-1125) never fired for 5s. Attribution driver-vs-QEMU-model is likely-but-not-pinned: source-side, rl_list_tx_init (:935-949) zeroes RL_TXADDR0-3 but NEVER clears RL_TXSTAT0-3, and rl_init does not rl_reset, so stale completion bits survive every re-init -- exactly the corner where the emulator's descriptor state and the driver desync; real silicon may be benign. Low severity: trigger is ifconfig (root), though the identical re-init state machine also runs unprivileged-adjacent via rl_watchdog/rl_intr.