DF-4365 — pcn_stop()/pcn_tick() AB-BA callout deadlock (DF-3715 family, pcn(4) instance)

File: sys/dev/netif/pcn/if_pcn.c · Severity: Medium · Class: CWE-667 improper locking / kernel wedge (DoS)

Bug (source)

pcn_stat_timer is a plain (non-MPSAFE) callout (callout_init at if_pcn.c:577) whose callback pcn_tick() does a blocking lwkt_serialize_enter(ifp->if_serializer) (if_pcn.c:879), self-re-arms at 1 Hz (callout_reset(..., hz, pcn_tick, sc) at if_pcn.c:896), and is torn down by callout_stop(&sc->pcn_stat_timer) in pcn_stop() (if_pcn.c:1329) — while every caller of pcn_stop() holds that same serializer: pcn_detach (:641-643), pcn_shutdown (:1374-1376), pcn_init (:1106, reached from pcn_ioctl SIOCSIFFLAGS :1273/:1276, pcn_watchdog :1306-1310, pcn_intr ERR :929), and pcn_intr directly (:912-914, interrupt context).

DragonFly's callout_stop() is synchronous: it ssleep()'s on 'costp' (kern_timeout.c:920) until an INPROG callback finishes (kern_timeout.c:567). A tick dequeued and parked in lwkt_serialize_enter() (lwkt_serialize.c:105) on the serializer held by the pcn_stop() caller therefore deadlocks against it (AB-BA): the stopper sleeps forever, the tick sleeps forever, that CPU's softclock stops delivering ALL callouts, and — because the callout is non-MPSAFE — the parked softclock thread holds the Giant MP lock while it sleeps.

Trigger on real hardware: ifconfig pcnX down (or up/down cycling, MTU/flags change, watchdog reinit, detach, suspend/resume, system shutdown) racing the 1 Hz tick.

Reproduce (guest, protocol replica — the guest has no PCnet NIC)

./build.sh     # ships + builds pcn_callout_replica{,_fixed}.ko on the guest
./run.sh       # loads replica, arms tick, fires pcn_stop shape in background

Expected (buggy protocol): serial console prints

pcnreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1)
pcnreplica[stop]: IF THIS IS THE LAST pcnreplica LINE callout_stop NEVER RETURNED (AB-BA deadlock)

and no further pcnreplica lines. Fresh ssh session shows the stopper in D0/wchan costp and one (softclock N) thread in B/wchan slize. The guest then fails to shut down cleanly.

Fix-shape validation (pcn_callout_replica_fixed.c, fresh guest): 308 try-enter retries during a 3 s serializer hold, callout_stop RETURNED, pending=0, guest healthy.

Afterwards: vm.sh reset with-src (the wedge survives clean shutdown).

Fix

fix.diff — make pcn_tick() non-blocking (lwkt_serialize_try() + 1-tick retry), the same shape validated for dc(4) in DF-3850.

DF-4365 — VERDICT

Status: inconclusive (driver-level runtime hardware-gated; mechanism PROVEN on the guest via deterministic protocol replica — DF-3715 family convention). Confidence: certain. Severity: Medium (permanent kernel wedge / local DoS on PCnet hardware).

What was proven, and how

The guest QEMU machine has a virtio-net NIC and no PCnet/PCI device, so the pcn(4) driver cannot be exercised at device level (pcn probe requires PCI vendor 0x1022 device 0x2000/0x2001 + Am79C97x CHIPID — if_pcn.c:101-107, :455-466). The bug, however, is not in the device logic: it is in the callout/serializer protocol pcn_stop()/pcn_tick() use, which is built entirely from generic DragonFly kernel primitives. Following the DF-3715 family methodology (DF-3715/3717/3738/3760/3776/3806/3850/3903/3906/3935/ 3991/4025/4046 — every one proven the same way), an out-of-tree module replicated the exact protocol:

• callout_init(&tick_c) — plain, like if_pcn.c:577 (NON-MPSAFE: callback executes under the Giant MP lock, kern_timeout.c:583);
• callback = blocking lwkt_serialize_enter(&ser) (park point, like if_pcn.c:879 via lwkt_serialize.c:105), then self re-arm (callout_reset(..., hz, ...), like if_pcn.c:896), then exit;
• stopper = a sysctl handler holding &ser while calling callout_stop(&tick_c) — the pcn_stop() shape (if_pcn.c:1329), in the serialized context every real caller provides (pcn_detach:641-643, pcn_shutdown:1374-1376, pcn_init:1106 ← pcn_ioctl:1273/1276, pcn_watchdog:1306-1310, pcn_intr:912-914/929).

Baseline run (buggy protocol) — run.log, boot.log.wedge

pcnreplica[stop]: tick INPROG and PARKED on the serializer (fired=2 ran=1)
pcnreplica[stop]: IF THIS IS THE LAST pcnreplica LINE callout_stop NEVER RETURNED (AB-BA deadlock)

Fresh ssh session (guest still pingable):

  1131 D0     costp    sysctl kern.pcnreplica_stop=1      <- stopper, kern_timeout.c:920
    -1 B2     slize    (softclock 2)                      <- parked tick, lwkt_serialize.c:105

D0/costp is precisely the synchronous callout_stop() wait; B2/slize is precisely the tick blocked on the serializer the stopper holds. No further pcnreplica console lines ever appeared (callout_stop never returned), CPU 2's softclock stopped delivering callouts, and the guest could not be shut down cleanly afterwards (vm.sh down had to SIGKILL qemu) — matching the family's "wedge survives shutdown" signature.

Fix-shape run (fixed protocol) — fix_validation.log

fix.diff gives pcn_tick() the lwkt_serialize_try() + 1-tick retry shape (validated for dc(4) in DF-3850). On a fresh guest, the fixed-shape replica was armed and the stopper held the serializer for 3 s from a background session (cross-CPU, like the baseline):

pcnrepfix[stop]: fires=309 ran=1 retries=308 - calling callout_stop
pcnrepfix[stop]: callout_stop RETURNED, pending=0
pcnrepfix[stat]: fires=310 ran=1 retries=309 pending=0

During the entire hold the tick fired 308 times, never parked (try-enter retries only), and callout_stop() returned with pending == 0. Guest healthy afterwards (uptime responsive). Deadlock eliminated at protocol level; driver-level validation is impossible on this guest (no PCnet NIC).

Incidental corroboration of the detach-race mode

An earlier iteration of the fixed replica allowed MOD_UNLOAD without stopping the armed callout; unloading with the tick armed produced Fatal trap 12 ... Stopped at softclock_handler+0xd9 — the armed callout executing unmapped module text. This is exactly what happens to pcn_tick if pcn_detach()'s teardown ever misses it, corroborating the post-detach UAF mode documented for the family (DF-3669/DF-3681) and showing the stop-side discipline is not optional.

Why the source says this is real for pcn(4) specifically

• if_pcn.c:577 — callout_init(&sc->pcn_stat_timer); (no _mp → non-MPSAFE).
• if_pcn.c:879 — lwkt_serialize_enter(ifp->if_serializer); (blocking).
• if_pcn.c:896 — callout_reset(&sc->pcn_stat_timer, hz, pcn_tick, sc); (unconditional self re-arm, under the serializer).
• if_pcn.c:1329 — callout_stop(&sc->pcn_stat_timer); inside pcn_stop().
• All pcn_stop() callers hold ifp->if_serializer at that point (see README). DragonFly callout_stop() waits synchronously for an INPROG callback (kern_timeout.c:920, :567), so the interleaving tick-parked-on-serializer vs stopper-in-callout_stop is a permanent AB-BA.

Verdict

The pcn(4) instance of the proven DF-3715 AB-BA callout family. On PCnet hardware, any ifconfig pcnX down/detach/shutdown/watchdog-reinit that races the 1 Hz pcn_tick permanently wedges two kernel threads, blacks out one CPU's callout wheel (with the Giant MP lock held by the parked softclock), and can make the system unable to shut down. The fix is the family fix: never block the softclock thread on the driver serializer (lwkt_serialize_try() + 1-tick retry), validated at protocol level on the guest (308 retries / 0 parks / callout_stop returned / pending=0).

Guest left clean (vm.sh reset with-src after both runs).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.