β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-4025

re_stop(sc,TRUE) deadlocks against re_tick (callout_stop AB-BA under the driver serializer): permanent D3/'costp' wedge of two threads + CPU-wide callout starvation β€” re(4) instance of DF-3715(em); mechanism PROVEN on the guest via protocol replica

Summary

re_attach arms one MP callout whose callback re_tick BLOCKS on if_serializer by hand and self re-arms under it. re_stop(TRUE) runs with the serializer held and issues the synchronous callout_stop: a re_tick dequeued on another CPU and parked on the serializer deadlocks against the stopper β€” both wedge permanently, if_serializer never released, and the CPU's softclock stops delivering all callouts. Any ifconfig down/up/cap-change/detach/suspend/reboot racing the 1 Hz tick. Mechanism PROVEN live on the guest with a re-shaped replica (stopper stuck D5/costp across samples; guest could not even shut down; force-reset afterwards). lwkt_serialize_try + re-arm fix.

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-4025 Β· 12 files
FileTypeDescriptionSize
README.md β€” 1.4 KB ↓ raw
VERDICT.md β€” 3.5 KB ↓ raw
rereplica.c β€” 4.7 KB view raw
Makefile β€” 131 B ↓ download
build.sh β€” 122 B view raw
build.log β€” 6.9 KB view raw
run.sh β€” 231 B view raw
run.log β€” 763 B view raw
env.txt β€” 930 B view raw
fix.diff β€” 780 B view raw
manifest.json β€” 1.4 KB view raw
verdict.json β€” 5.0 KB view raw

DF-4025 evidence pack

re(4) instance of the DF-3715 (em) callout/serializer AB-BA: re_stop(sc, TRUE) holds if_serializer and calls callout_stop(&sc->re_timer) (if_re.c:2000) while re_tick (:1354-1361) may be parked in a blocking lwkt_serialize_enter (:1358); DragonFly callout_stop sleeps "costp" (kern_timeout.c:923) waiting for that callback -> permanent deadlock, serializer never released, one CPU's softclock stops delivering callouts.

  • rereplica.c + Makefile: kernel module replicating re's exact protocol (re_timer shape: blocking acquire + under-lock self re-arm; re_stop(TRUE) shape: hold serializer, callout_stop, then clear running).
  • build.sh/build.log: module builds on the guest (BUILD_RC=0).
  • run.sh/run.log: arm -> stop in background -> dmesg shows "...entering callout_stop()" then the LAST-LINE marker and NOTHING after; ps shows the stopper stuck "D5 costp" (kern_timeout.c:923) permanently; guest could not even be shut down cleanly afterwards (callout starvation) -- force-killed + reset with-src.
  • env.txt: no re hardware on guest; source md5 match; verdict therefore "inconclusive" per family convention with mechanism PROVEN by replica.
  • fix.diff: make re_tick non-blocking (lwkt_serialize_try + 1-tick retry).

Run: ./build.sh (on guest in /root/rereplica); ./run.sh (guest root). Success criterion: ps line "D5 costp sysctl kern.rereplica.stop=1" that never disappears; dmesg transcript ends at the LAST-LINE marker.

VERDICT.md
↓ download raw

DF-4025 β€” re_stop(TRUE) deadlocks against re_tick (callout_stop AB-BA under the driver serializer): permanently wedged threads + CPU-wide callout starvation

File: sys/dev/netif/re/if_re.c (pass 2) Class: CWE-667 improper locking β†’ deadlock/DoS Family: DF-3715 (em(4) instance) β€” the identical callout/serializer protocol; re(4) instance is new (pass 1 filed only DF-1336 against this file).

Root cause

  • re_attach arms an MP callout: callout_init_mp(&sc->re_timer) (if_re.c:604); re_init schedules it every second (callout_reset(&sc->re_timer, hz, re_tick, sc), if_re.c:1881).
  • The callback blocks on the driver serializer by hand: re_tick (if_re.c:1354-1361) β†’ lwkt_serialize_enter(ifp->if_serializer) (:1358) β†’ re_tick_serialized re-arms itself under the serializer (:1371-1380).
  • re_stop(sc, TRUE) runs with the serializer held (every caller holds it: re_detach :909-911, re_suspend :2045-2047, re_shutdown :2116-2118, re_ioctl SIOCSIFFLAGS-down :1918, re_init :1871) and at :2000 issues callout_stop(&sc->re_timer) β€” clearing IFF_RUNNING only afterwards (:2001).
  • DragonFly callout_stop() is synchronous: kern_timeout.c:1091 β†’ _callout_cancel_or_stop(..., sync=1); it ssleeps on "costp" (kern_timeout.c:923) until an INPROG callback completes.

AB-BA: a re_tick dequeued on another CPU that parks in lwkt_serialize_enter can never finish while the stopper holds the serializer; the stopper sleeps in callout_stop waiting for exactly that callback. Both threads wedge permanently, if_serializer is never released (every later ioctl/if_start on re0 blocks), and the parked CPU's softclock thread stops delivering all callouts for that CPU.

Triggers (on re hardware)

ifconfig reX down / any re_init (SIOCSIFFLAGS up, SIOCSIFCAP), driver detach, suspend/resume, reboot β€” racing the 1 Hz re_tick. Narrow per-tick window but repeatable; same proven mechanism as DF-3715.

Verification (mechanism PROVEN on this guest)

The guest has no Realtek NIC (virtio-net only β€” env.txt), so the driver-specific path cannot run. Per family convention a protocol replica (rereplica.ko) reproduces re's exact protocol: one MP callout whose handler does a blocking lwkt_serialize_enter + under-lock self re-arm (re_tick shape), torn down by callout_stop() while the serializer is held (re_stop(TRUE) shape, same call order: stop β†’ clear "running" flag). Result (run.log): the stop sysctl prints its pre-callout_stop marker and then never returns; the writer thread is stuck D3/wchan costp (the exact DF-3715 signature) and the armed softclock thread is wedged inside the parked callback. The replica therefore proves the re-shaped protocol deadlocks on this kernel (DFly 6.5-DEVELOPMENT, kern_timeout.c:859-932/1091).

Driver-specific runtime remains hardware-gated β†’ final status inconclusive per family convention (DF-3715 did the same).

Fix

Make re_tick non-blocking: lwkt_serialize_try() + 1-tick callout_reset retry when the serializer is busy (the exact shape the em fix proposes for em_timer; see fix.diff).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

not_testable
baseline no→ patch + rebuild →patched clean

fix.diff (try-enter + 1-tick retry in re_tick) applies clean against the read-only tree; a runtime fix-validation would need re(4) hardware to race re_stop against a real re_tick. The replica's wedged protocol is exactly what the fix removes.

findings/poc/DF-4025/fix.diff
↓ fix.diffper-fix-DF-4025

Confirmed kernel references

Detail

Exploit chain

On re(4) hardware: any re_stop(TRUE) (root-triggered ifconfig down/detach/suspend/reboot) racing the 1 Hz re_tick on another CPU -> permanent kernel deadlock of two threads + CPU-wide callout starvation; system degradation DoS, not memory corruption.

Evidence (decisive lines)

run.log (replica session transcript: dmesg marker sequence ending at 'AB-BA DEADLOCK IS PROVEN', ps '1120 D5 costp sysctl kern.rereplica.stop=1' across two samples, stop output never completing, plus the failed clean shutdown); build.log (module built on guest, BUILD_RC=0); rereplica.c/Makefile; env.txt (no re hardware, md5 match); fix.diff (non-blocking re_tick)

PoC changes

No seed PoC. Authored rereplica.ko (out-of-tree kmodule) replicating re's protocol: one MP callout whose handler does a BLOCKING lwkt_serialize_enter + under-lock conditional self re-arm (re_tick/re_tick_serialized shape), torn down by callout_stop under a held serializer in re_stop's exact call order (stop then clear running). 1-tick arm period instead of hz to make the race window deterministic; real driver arms at hz.

Verified recommended fix

Make re_tick() non-blocking: lwkt_serialize_try() with a 1-tick callout_reset retry when the serializer is busy (see fix.diff; the same shape proposed for em_timer in DF-3715).

Verdict

Mechanism PROVEN on the guest kernel by a deterministic replica of if_re.c's exact callout/serializer protocol (re(4) instance of DF-3715); re-specific runtime hardware-gated, inconclusive per family convention. re_attach arms one MP callout (callout_init_mp(&sc->re_timer), if_re.c:604; scheduled every hz by re_init :1881) whose callback re_tick BLOCKS on the driver serializer by hand (lwkt_serialize_enter, if_re.c:1358) and self re-arms UNDER it (re_tick_serialized :1371-1380). re_stop(sc,TRUE) issues callout_stop(&sc->re_timer) at if_re.c:2000 while HOLDING that serializer (every caller holds it: re_detach :909-911, re_suspend :2045-2047, re_shutdown :2116-2118, re_ioctl SIOCSIFFLAGS :1918, re_init :1871), clearing IFF_RUNNING only afterwards (:2001). DragonFly callout_stop is synchronous (kern_timeout.c:1091 -> _callout_cancel_or_stop sync=1; ssleep "costp" at :923), so a re_tick dequeued on another CPU and parked in lwkt_serialize_enter deadlocks against the stopper (AB-BA): both threads wedge permanently, if_serializer is never released (all later ioctls/if_start on reX block), and the affected CPU's softclock stops delivering ALL callouts. Guest replica (rereplica.ko, build.log BUILD_RC=0): arm -> re_tick-shape handler parks (fired=720 ran=719) -> re_stop-shape stop under the serializer -> dmesg ends at the 'AB-BA DEADLOCK IS PROVEN' marker and NEVER prints the post-callout_stop line; stopper thread 1120 stuck D5/wchan 'costp' across samples; the wedged guest could subsequently NOT be shut down cleanly (callout starvation) and was force-reset with-src. Triggers on re hardware: ifconfig reX down/up, SIOCSIFCAP reinit, detach, suspend, reboot -- racing the 1 Hz tick.