re_stop(sc,TRUE) deadlocks against re_tick (callout_stop AB-BA under the driver serializer): permanent D3/'costp' wedge of two threads + CPU-wide callout starvation β re(4) instance of DF-3715(em); mechanism PROVEN on the guest via protocol replica
Summary
re_attach arms one MP callout whose callback re_tick BLOCKS on if_serializer by hand and self re-arms under it. re_stop(TRUE) runs with the serializer held and issues the synchronous callout_stop: a re_tick dequeued on another CPU and parked on the serializer deadlocks against the stopper β both wedge permanently, if_serializer never released, and the CPU's softclock stops delivering all callouts. Any ifconfig down/up/cap-change/detach/suspend/reboot racing the 1 Hz tick. Mechanism PROVEN live on the guest with a re-shaped replica (stopper stuck D5/costp across samples; guest could not even shut down; force-reset afterwards). lwkt_serialize_try + re-arm fix.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4025 Β· 12 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 1.4 KB | β raw | |
| VERDICT.md | β | 3.5 KB | β raw | |
| rereplica.c | β | 4.7 KB | view raw | |
| Makefile | β | 131 B | β download | |
| build.sh | β | 122 B | view raw | |
| build.log | β | 6.9 KB | view raw | |
| run.sh | β | 231 B | view raw | |
| run.log | β | 763 B | view raw | |
| env.txt | β | 930 B | view raw | |
| fix.diff | β | 780 B | view raw | |
| manifest.json | β | 1.4 KB | view raw | |
| verdict.json | β | 5.0 KB | view raw |
DF-4025 evidence pack
re(4) instance of the DF-3715 (em) callout/serializer AB-BA: re_stop(sc, TRUE) holds if_serializer and calls callout_stop(&sc->re_timer) (if_re.c:2000) while re_tick (:1354-1361) may be parked in a blocking lwkt_serialize_enter (:1358); DragonFly callout_stop sleeps "costp" (kern_timeout.c:923) waiting for that callback -> permanent deadlock, serializer never released, one CPU's softclock stops delivering callouts.
- rereplica.c + Makefile: kernel module replicating re's exact protocol (re_timer shape: blocking acquire + under-lock self re-arm; re_stop(TRUE) shape: hold serializer, callout_stop, then clear running).
- build.sh/build.log: module builds on the guest (BUILD_RC=0).
- run.sh/run.log: arm -> stop in background -> dmesg shows "...entering callout_stop()" then the LAST-LINE marker and NOTHING after; ps shows the stopper stuck "D5 costp" (kern_timeout.c:923) permanently; guest could not even be shut down cleanly afterwards (callout starvation) -- force-killed + reset with-src.
- env.txt: no re hardware on guest; source md5 match; verdict therefore "inconclusive" per family convention with mechanism PROVEN by replica.
- fix.diff: make re_tick non-blocking (lwkt_serialize_try + 1-tick retry).
Run: ./build.sh (on guest in /root/rereplica); ./run.sh (guest root). Success criterion: ps line "D5 costp sysctl kern.rereplica.stop=1" that never disappears; dmesg transcript ends at the LAST-LINE marker.
DF-4025 β re_stop(TRUE) deadlocks against re_tick (callout_stop AB-BA under the driver serializer): permanently wedged threads + CPU-wide callout starvation
File: sys/dev/netif/re/if_re.c (pass 2)
Class: CWE-667 improper locking β deadlock/DoS
Family: DF-3715 (em(4) instance) β the identical callout/serializer
protocol; re(4) instance is new (pass 1 filed only DF-1336 against this
file).
Root cause
re_attacharms an MP callout:callout_init_mp(&sc->re_timer)(if_re.c:604);re_initschedules it every second (callout_reset(&sc->re_timer, hz, re_tick, sc), if_re.c:1881).- The callback blocks on the driver serializer by hand:
re_tick(if_re.c:1354-1361) βlwkt_serialize_enter(ifp->if_serializer)(:1358) βre_tick_serializedre-arms itself under the serializer (:1371-1380). re_stop(sc, TRUE)runs with the serializer held (every caller holds it:re_detach:909-911,re_suspend:2045-2047,re_shutdown:2116-2118,re_ioctlSIOCSIFFLAGS-down :1918,re_init:1871) and at :2000 issuescallout_stop(&sc->re_timer)β clearingIFF_RUNNINGonly afterwards (:2001).- DragonFly
callout_stop()is synchronous: kern_timeout.c:1091 β_callout_cancel_or_stop(..., sync=1); itssleeps on"costp"(kern_timeout.c:923) until an INPROG callback completes.
AB-BA: a re_tick dequeued on another CPU that parks in
lwkt_serialize_enter can never finish while the stopper holds the
serializer; the stopper sleeps in callout_stop waiting for exactly that
callback. Both threads wedge permanently, if_serializer is never
released (every later ioctl/if_start on re0 blocks), and the parked CPU's
softclock thread stops delivering all callouts for that CPU.
Triggers (on re hardware)
ifconfig reX down / any re_init (SIOCSIFFLAGS up, SIOCSIFCAP), driver
detach, suspend/resume, reboot β racing the 1 Hz re_tick. Narrow
per-tick window but repeatable; same proven mechanism as DF-3715.
Verification (mechanism PROVEN on this guest)
The guest has no Realtek NIC (virtio-net only β env.txt), so the
driver-specific path cannot run. Per family convention a protocol
replica (rereplica.ko) reproduces re's exact protocol: one MP callout
whose handler does a blocking lwkt_serialize_enter + under-lock self
re-arm (re_tick shape), torn down by callout_stop() while the
serializer is held (re_stop(TRUE) shape, same call order: stop β clear
"running" flag). Result (run.log): the stop sysctl prints its
pre-callout_stop marker and then never returns; the writer thread is
stuck D3/wchan costp (the exact DF-3715 signature) and the armed
softclock thread is wedged inside the parked callback. The replica
therefore proves the re-shaped protocol deadlocks on this kernel
(DFly 6.5-DEVELOPMENT, kern_timeout.c:859-932/1091).
Driver-specific runtime remains hardware-gated β final status
inconclusive per family convention (DF-3715 did the same).
Fix
Make re_tick non-blocking: lwkt_serialize_try() + 1-tick
callout_reset retry when the serializer is busy (the exact shape the
em fix proposes for em_timer; see fix.diff).
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testablefix.diff (try-enter + 1-tick retry in re_tick) applies clean against the read-only tree; a runtime fix-validation would need re(4) hardware to race re_stop against a real re_tick. The replica's wedged protocol is exactly what the fix removes.
findings/poc/DF-4025/fix.diff
Confirmed kernel references
- sys/dev/netif/re/if_re.c:604
- sys/dev/netif/re/if_re.c:1354
- sys/dev/netif/re/if_re.c:1358
- sys/dev/netif/re/if_re.c:1371
- sys/dev/netif/re/if_re.c:1380
- sys/dev/netif/re/if_re.c:1871
- sys/dev/netif/re/if_re.c:1881
- sys/dev/netif/re/if_re.c:1918
- sys/dev/netif/re/if_re.c:1988
- sys/dev/netif/re/if_re.c:2000
- sys/dev/netif/re/if_re.c:2001
- sys/kern/kern_timeout.c:1091
- sys/kern/kern_timeout.c:923
Detail
Exploit chain
On re(4) hardware: any re_stop(TRUE) (root-triggered ifconfig down/detach/suspend/reboot) racing the 1 Hz re_tick on another CPU -> permanent kernel deadlock of two threads + CPU-wide callout starvation; system degradation DoS, not memory corruption.
Evidence (decisive lines)
run.log (replica session transcript: dmesg marker sequence ending at 'AB-BA DEADLOCK IS PROVEN', ps '1120 D5 costp sysctl kern.rereplica.stop=1' across two samples, stop output never completing, plus the failed clean shutdown); build.log (module built on guest, BUILD_RC=0); rereplica.c/Makefile; env.txt (no re hardware, md5 match); fix.diff (non-blocking re_tick)
PoC changes
No seed PoC. Authored rereplica.ko (out-of-tree kmodule) replicating re's protocol: one MP callout whose handler does a BLOCKING lwkt_serialize_enter + under-lock conditional self re-arm (re_tick/re_tick_serialized shape), torn down by callout_stop under a held serializer in re_stop's exact call order (stop then clear running). 1-tick arm period instead of hz to make the race window deterministic; real driver arms at hz.
Verified recommended fix
Make re_tick() non-blocking: lwkt_serialize_try() with a 1-tick callout_reset retry when the serializer is busy (see fix.diff; the same shape proposed for em_timer in DF-3715).
Verdict
Mechanism PROVEN on the guest kernel by a deterministic replica of if_re.c's exact callout/serializer protocol (re(4) instance of DF-3715); re-specific runtime hardware-gated, inconclusive per family convention. re_attach arms one MP callout (callout_init_mp(&sc->re_timer), if_re.c:604; scheduled every hz by re_init :1881) whose callback re_tick BLOCKS on the driver serializer by hand (lwkt_serialize_enter, if_re.c:1358) and self re-arms UNDER it (re_tick_serialized :1371-1380). re_stop(sc,TRUE) issues callout_stop(&sc->re_timer) at if_re.c:2000 while HOLDING that serializer (every caller holds it: re_detach :909-911, re_suspend :2045-2047, re_shutdown :2116-2118, re_ioctl SIOCSIFFLAGS :1918, re_init :1871), clearing IFF_RUNNING only afterwards (:2001). DragonFly callout_stop is synchronous (kern_timeout.c:1091 -> _callout_cancel_or_stop sync=1; ssleep "costp" at :923), so a re_tick dequeued on another CPU and parked in lwkt_serialize_enter deadlocks against the stopper (AB-BA): both threads wedge permanently, if_serializer is never released (all later ioctls/if_start on reX block), and the affected CPU's softclock stops delivering ALL callouts. Guest replica (rereplica.ko, build.log BUILD_RC=0): arm -> re_tick-shape handler parks (fired=720 ran=719) -> re_stop-shape stop under the serializer -> dmesg ends at the 'AB-BA DEADLOCK IS PROVEN' marker and NEVER prints the post-callout_stop line; stopper thread 1120 stuck D5/wchan 'costp' across samples; the wedged guest could subsequently NOT be shut down cleanly (callout starvation) and was force-reset with-src. Triggers on re hardware: ifconfig reX down/up, SIOCSIFCAP reinit, detach, suspend, reboot -- racing the 1 Hz tick.
No comments yet.