β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-4801

Revive waitlist is orphaned when a revive stops: queued conflicting writes hang forever in uninterruptible biowait β€” kill -9-proof, survives 'vinum setstate -f down' abort, leaks the requests

Summary

A write straddling the revive cursor queues on sd->waitlist and sleeps PCATCH-less; the ONLY drain in the tree is the tail of a successful revive_block, driven one block per ioctl by userland - when that process stops, no state transition/detach/unload ever drains. REPRODUCED deterministic on stock kernel: one priming ioctl + 4 writers on the straddle window -> 4x D-state dd, kill -9 ineffective, STILL HUNG after explicit abort; requests leak until reboot and pin the module. drain_sd_waitlist-on-transition fix validated (stacks race-safe on the DF-4482 v2 cmpset).

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-4801 Β· 10 files
FileTypeDescriptionSize
hang.sh β€” 3.0 KB view raw
prime.c β€” 1.6 KB view raw
run.log β€” 2.7 KB view raw
fix.diff β€” 2.2 KB view raw
fix_run.log β€” 1.6 KB view raw
env.txt β€” 312 B view raw
README.md β€” 3.6 KB ↓ raw
VERDICT.md β€” 2.6 KB ↓ raw
verdict.json β€” 3.6 KB view raw
manifest.json β€” 970 B view raw

DF-4801 β€” the revive waitlist is never drained when a revive is abandoned: writers queue on sd->waitlist forever (uninterruptible, kill -9-proof, survives revive abort/detach)

  • File: sys/dev/raid/vinum/vinumrevive.c:198-218 (the only drain in the tree), vinumrequest.c:327-355 (the append)
  • Severity: Medium (permanent unkillable I/O hang + request leak; root starts the revive, unprivileged writers via a mounted filesystem are the victims; only a reboot clears it)
  • Bucket: base:raid / dos
  • Status: reproduced (4/4 writers hung, deterministic), fix validated on the guest

Root cause

A write that straddles the revive cursor is queued on sd->waitlist (launch_requests(), XFR_REVIVECONFLICT, vinumrequest.c:327-355) and its writer sleeps in biowait() β€” which has no PCATCH, so the sleep is uninterruptible. The ONLY code in the whole tree that drains sd->waitlist is the tail of a successful revive_block() call (vinumrevive.c:198-218). The revive itself is driven one block per ioctl by the userland vinum start process; when that process stops issuing ioctls β€” killed, SIGSTOPped, crashed, or simply not scheduled β€” the waitlist is orphaned:

  • no set_sd_state() transition away from sd_reviving (down/stale/initializing/…) looks at sd->waitlist (vinumstate.c:104-296);
  • subdisk detach / vinum stop / module unload never drain it (rg waitlist over sys/dev/raid/vinum/ shows exactly one drain site);
  • the queued struct requests and their buffers leak permanently.

Result: every writer whose request was queued hangs forever, is immune to kill -9, and stays hung through an explicit revive abort (vinum setstate -f down).

Reproduction (guest, stock kernel, deterministic β€” no UAF race needed)

prime.c sends exactly ONE VINUM_SETSTATE(object_up, blocksize=64k) ioctl: one clean revive_block() runs (empty waitlist), the sd is now sd_reviving with the cursor at byte 64k, and nothing will ever drive it again. Four writers then hammer the straddle window [64k, 96k) (every write conflicts β†’ waitlisted). hang.sh:

prime: error=35 msg=                       <- EAGAIN, sd "R 2%"
writer1..4: (frozen immediately, counters empty)
 918 D0     physstr  dd ... seek=18 ...    <- all four writers D-state
kill -9 -> all four STILL D-state          (">>> UNKILLABLE HUNG WRITERS <<<")
vinum setstate -f down vol.p1.s0
 -> all four STILL D-state                 (">>> STILL HUNG AFTER ABORT <<<")
S vol.p1.s0   State: down                  (revive aborted; nobody drained)

Writers target only the straddle window so the trigger never touches the REQUEST_DOWN / trailing-BAD-group launch path (the DF-4482/DF-4721 UAF) β€” the hang is shown in isolation, deterministically (run.log).

Fix (validated)

fix.diff adds drain_sd_waitlist(sdno) (steal the list atomically; relaunch each request if the sd just came up β€” exactly what revive_block()'s own drain would have done β€” otherwise fail it with EIO and free it), called from set_sd_state() on every actual transition out of sd_reviving (vinumstate.c hook). Aborting the revive now completes the queued writes:

-- after revive abort --
writers gone (drained: fix working)        (fix_run.log)

vs. the stock kernel's >>> STILL HUNG AFTER ABORT <<<.

Files

  • hang.sh β€” the exact trigger (setup + prime + 4 straddle writers + kill -9 tests + abort test)
  • prime.c β€” single-ioctl revive primer
  • run.log β€” stock reproduction
  • fix.diff β€” drain-on-abort (validated), stacks on the DF-4482 v2 fix
  • fix_run.log β€” patched-module validation
  • env.txt
VERDICT.md
↓ download raw

DF-4801 verdict β€” REPRODUCED (permanent unkillable writer hang, deterministic), fix validated

  • finding: sd->waitlist is drained ONLY from the tail of a successful revive_block() (sys/dev/raid/vinum/vinumrevive.c:198-218). The revive is driven one block per ioctl by userland vinum start; when that stops (killed, SIGSTOP, crash, scheduler starvation), queued conflicting writes (launch_requests(), vinumrequest.c:327-355) sleep in PCATCH-less biowait forever. No state transition (set_sd_state, detach, stop, unload) drains the list.
  • guest: DragonFly dfbsd 6.5-DEVELOPMENT #0 (stock), mirrored concat volume on md drives.

Runs (stock)

One VINUM_SETSTATE(object_up, blocksize=64k) ioctl primes the sd into sd_reviving (cursor at 64k, empty waitlist, no drainer ever again); four writers hammer [64k,96k) β€” every write is XFR_REVIVECONFLICT and queues:

  • all four writers freeze instantly, D-state (wchan physstr);
  • kill -9 on writers AND their dd children: all four remain;
  • vinum setstate -f down vol.p1.s0 (explicit revive abort): all four remain (>>> STILL HUNG AFTER ABORT <<<, run.log).

Deterministic (no race needed): the writers' requests are queued the moment they straddle the frozen cursor; the trigger deliberately avoids the DF-4482/DF-4721 launch paths so the hang shows in isolation. 4/4 writers hung; the requests and their buffers leak until reboot.

Unprivileged angle: any filesystem mounted on the volume turns every user write that straddles a frozen/abandoned cursor into a permanent unkillable hang of that user's process.

Fix validation

fix.diff: drain_sd_waitlist() (atomic steal; relaunch if the sd came up, else biodone(EIO)+freerq) hooked into set_sd_state() on every real transition out of sd_reviving. Patched module, same workload: writers still freeze while queued (expected), and the abort drains them β€” post-abort ps shows no dd left (fix_run.log), vs. stock's four immortal D-state writers.

fix_status: fixed (baseline_reproduced=1, patched_reproduced=0).

Residual (documented in fix.diff): transitions that bypass set_sd_state's commit point (e.g. the driveoffset < 0 early path, direct slot reuse) are not covered; a full redesign would keep the waitlist in the sd's own teardown.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

patched module: identical workload, writers queue as before while the revive is frozen, and the 'setstate -f down' abort now drains them - post-abort ps shows no dd/writers left (fix_run.log), vs stock where all four remained D-state indefinitely

fix_run.log
↓ fix.diffper-fix-DF-4801

Confirmed kernel references

Detail

Exploit chain

root: mirrored volume + revive started then stopped (kill the 'vinum start' process mid-revive, or let one VINUM_SETSTATE(object_up) ioctl prime sd_reviving and never drive it again); unprivileged: any write straddling the frozen cursor through a mounted filesystem queues on sd->waitlist -> uninterruptible biowait -> permanent unkillable hang, survives kill -9 and revive abort, only reboot clears it

Evidence (decisive lines)

run.log (stock: 4/4 writers D-state, '>>> UNKILLABLE HUNG WRITERS <<<' after kill -9 and '>>> STILL HUNG AFTER ABORT <<<' after setstate -f down); fix_run.log (patched: 'writers gone (drained: fix working)')

PoC changes

wrote prime.c (single-ioctl revive primer) because userland 'vinum setstate -f up' loops the ioctl internally until the revive completes; writers restricted to the straddle window [64k,96k) so the trigger never touches the REQUEST_DOWN/trailing-BAD launch path (DF-4721 arm) and shows the hang deterministically

Verified recommended fix

drain_sd_waitlist() on every set_sd_state() transition out of sd_reviving: relaunch queued requests when the sd comes up, biodone(EIO)+freerq otherwise (fix.diff, validated)

Verdict

sd->waitlist is drained ONLY from the tail of a successful revive_block() (vinumrevive.c:198-218; the sole drain site in the tree). The revive is driven one block per ioctl by userland 'vinum start'; when that process stops (killed, SIGSTOP, crash), conflicting writes queued on the waitlist (vinumrequest.c:327-355) sleep in PCATCH-less biowait forever. Reproduced deterministically on the stock kernel: one VINUM_SETSTATE(object_up) ioctl primes sd_reviving with the cursor at 64k; four writers straddling [64k,96k) all queue; all four freeze in D-state, survive kill -9 of writers and dd children, and REMAIN hung after an explicit 'vinum setstate -f down' revive abort (run.log). The queued struct requests and buffers leak until reboot. With a filesystem mounted on the volume, the victims are unprivileged writers. Fix (drain_sd_waitlist() hooked into set_sd_state() on every real transition out of sd_reviving: relaunch if the sd came up, biodone(EIO)+freerq otherwise) validated on the same guest: post-abort ps shows no hung writers (fix_run.log) vs stock's four immortal D-state dd's.