Revive waitlist is orphaned when a revive stops: queued conflicting writes hang forever in uninterruptible biowait β kill -9-proof, survives 'vinum setstate -f down' abort, leaks the requests
Summary
A write straddling the revive cursor queues on sd->waitlist and sleeps PCATCH-less; the ONLY drain in the tree is the tail of a successful revive_block, driven one block per ioctl by userland - when that process stops, no state transition/detach/unload ever drains. REPRODUCED deterministic on stock kernel: one priming ioctl + 4 writers on the straddle window -> 4x D-state dd, kill -9 ineffective, STILL HUNG after explicit abort; requests leak until reboot and pin the module. drain_sd_waitlist-on-transition fix validated (stacks race-safe on the DF-4482 v2 cmpset).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-4801 Β· 10 files| File | Type | Description | Size | |
|---|---|---|---|---|
| hang.sh | β | 3.0 KB | view raw | |
| prime.c | β | 1.6 KB | view raw | |
| run.log | β | 2.7 KB | view raw | |
| fix.diff | β | 2.2 KB | view raw | |
| fix_run.log | β | 1.6 KB | view raw | |
| env.txt | β | 312 B | view raw | |
| README.md | β | 3.6 KB | β raw | |
| VERDICT.md | β | 2.6 KB | β raw | |
| verdict.json | β | 3.6 KB | view raw | |
| manifest.json | β | 970 B | view raw |
DF-4801 β the revive waitlist is never drained when a revive is abandoned: writers queue on sd->waitlist forever (uninterruptible, kill -9-proof, survives revive abort/detach)
- File:
sys/dev/raid/vinum/vinumrevive.c:198-218(the only drain in the tree),vinumrequest.c:327-355(the append) - Severity: Medium (permanent unkillable I/O hang + request leak; root starts the revive, unprivileged writers via a mounted filesystem are the victims; only a reboot clears it)
- Bucket: base:raid / dos
- Status: reproduced (4/4 writers hung, deterministic), fix validated on the guest
Root cause
A write that straddles the revive cursor is queued on sd->waitlist
(launch_requests(), XFR_REVIVECONFLICT, vinumrequest.c:327-355)
and its writer sleeps in biowait() β which has no PCATCH, so the
sleep is uninterruptible. The ONLY code in the whole tree that drains
sd->waitlist is the tail of a successful revive_block() call
(vinumrevive.c:198-218). The revive itself is driven one block per
ioctl by the userland vinum start process; when that process stops
issuing ioctls β killed, SIGSTOPped, crashed, or simply not scheduled β
the waitlist is orphaned:
- no
set_sd_state()transition away fromsd_reviving(down/stale/initializing/β¦) looks atsd->waitlist(vinumstate.c:104-296); - subdisk detach /
vinum stop/ module unload never drain it (rg waitlistoversys/dev/raid/vinum/shows exactly one drain site); - the queued
struct requests and their buffers leak permanently.
Result: every writer whose request was queued hangs forever, is
immune to kill -9, and stays hung through an explicit revive abort
(vinum setstate -f down).
Reproduction (guest, stock kernel, deterministic β no UAF race needed)
prime.c sends exactly ONE VINUM_SETSTATE(object_up, blocksize=64k)
ioctl: one clean revive_block() runs (empty waitlist), the sd is now
sd_reviving with the cursor at byte 64k, and nothing will ever drive
it again. Four writers then hammer the straddle window [64k, 96k)
(every write conflicts β waitlisted). hang.sh:
prime: error=35 msg= <- EAGAIN, sd "R 2%"
writer1..4: (frozen immediately, counters empty)
918 D0 physstr dd ... seek=18 ... <- all four writers D-state
kill -9 -> all four STILL D-state (">>> UNKILLABLE HUNG WRITERS <<<")
vinum setstate -f down vol.p1.s0
-> all four STILL D-state (">>> STILL HUNG AFTER ABORT <<<")
S vol.p1.s0 State: down (revive aborted; nobody drained)
Writers target only the straddle window so the trigger never touches
the REQUEST_DOWN / trailing-BAD-group launch path (the DF-4482/DF-4721
UAF) β the hang is shown in isolation, deterministically (run.log).
Fix (validated)
fix.diff adds drain_sd_waitlist(sdno) (steal the list atomically;
relaunch each request if the sd just came up β exactly what
revive_block()'s own drain would have done β otherwise fail it with
EIO and free it), called from set_sd_state() on every actual
transition out of sd_reviving (vinumstate.c hook). Aborting the
revive now completes the queued writes:
-- after revive abort -- writers gone (drained: fix working) (fix_run.log)
vs. the stock kernel's >>> STILL HUNG AFTER ABORT <<<.
Files
hang.shβ the exact trigger (setup + prime + 4 straddle writers + kill -9 tests + abort test)prime.cβ single-ioctl revive primerrun.logβ stock reproductionfix.diffβ drain-on-abort (validated), stacks on the DF-4482 v2 fixfix_run.logβ patched-module validationenv.txt
DF-4801 verdict β REPRODUCED (permanent unkillable writer hang, deterministic), fix validated
- finding:
sd->waitlistis drained ONLY from the tail of a successfulrevive_block()(sys/dev/raid/vinum/vinumrevive.c:198-218). The revive is driven one block per ioctl by userlandvinum start; when that stops (killed, SIGSTOP, crash, scheduler starvation), queued conflicting writes (launch_requests(),vinumrequest.c:327-355) sleep in PCATCH-lessbiowaitforever. No state transition (set_sd_state, detach, stop, unload) drains the list. - guest: DragonFly dfbsd 6.5-DEVELOPMENT #0 (stock), mirrored concat volume on md drives.
Runs (stock)
One VINUM_SETSTATE(object_up, blocksize=64k) ioctl primes the sd into
sd_reviving (cursor at 64k, empty waitlist, no drainer ever again);
four writers hammer [64k,96k) β every write is XFR_REVIVECONFLICT and
queues:
- all four writers freeze instantly,
D-state (wchan physstr); kill -9on writers AND their dd children: all four remain;vinum setstate -f down vol.p1.s0(explicit revive abort): all four remain (>>> STILL HUNG AFTER ABORT <<<,run.log).
Deterministic (no race needed): the writers' requests are queued the moment they straddle the frozen cursor; the trigger deliberately avoids the DF-4482/DF-4721 launch paths so the hang shows in isolation. 4/4 writers hung; the requests and their buffers leak until reboot.
Unprivileged angle: any filesystem mounted on the volume turns every user write that straddles a frozen/abandoned cursor into a permanent unkillable hang of that user's process.
Fix validation
fix.diff: drain_sd_waitlist() (atomic steal; relaunch if the sd came
up, else biodone(EIO)+freerq) hooked into set_sd_state() on every
real transition out of sd_reviving. Patched module, same workload:
writers still freeze while queued (expected), and the abort drains them
β post-abort ps shows no dd left (fix_run.log), vs. stock's four
immortal D-state writers.
fix_status: fixed (baseline_reproduced=1, patched_reproduced=0).
Residual (documented in fix.diff): transitions that bypass
set_sd_state's commit point (e.g. the driveoffset < 0 early path,
direct slot reuse) are not covered; a full redesign would keep the
waitlist in the sd's own teardown.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedpatched module: identical workload, writers queue as before while the revive is frozen, and the 'setstate -f down' abort now drains them - post-abort ps shows no dd/writers left (fix_run.log), vs stock where all four remained D-state indefinitely
fix_run.log
Confirmed kernel references
Detail
Exploit chain
root: mirrored volume + revive started then stopped (kill the 'vinum start' process mid-revive, or let one VINUM_SETSTATE(object_up) ioctl prime sd_reviving and never drive it again); unprivileged: any write straddling the frozen cursor through a mounted filesystem queues on sd->waitlist -> uninterruptible biowait -> permanent unkillable hang, survives kill -9 and revive abort, only reboot clears it
Evidence (decisive lines)
run.log (stock: 4/4 writers D-state, '>>> UNKILLABLE HUNG WRITERS <<<' after kill -9 and '>>> STILL HUNG AFTER ABORT <<<' after setstate -f down); fix_run.log (patched: 'writers gone (drained: fix working)')
PoC changes
wrote prime.c (single-ioctl revive primer) because userland 'vinum setstate -f up' loops the ioctl internally until the revive completes; writers restricted to the straddle window [64k,96k) so the trigger never touches the REQUEST_DOWN/trailing-BAD launch path (DF-4721 arm) and shows the hang deterministically
Verified recommended fix
drain_sd_waitlist() on every set_sd_state() transition out of sd_reviving: relaunch queued requests when the sd comes up, biodone(EIO)+freerq otherwise (fix.diff, validated)
Verdict
sd->waitlist is drained ONLY from the tail of a successful revive_block() (vinumrevive.c:198-218; the sole drain site in the tree). The revive is driven one block per ioctl by userland 'vinum start'; when that process stops (killed, SIGSTOP, crash), conflicting writes queued on the waitlist (vinumrequest.c:327-355) sleep in PCATCH-less biowait forever. Reproduced deterministically on the stock kernel: one VINUM_SETSTATE(object_up) ioctl primes sd_reviving with the cursor at 64k; four writers straddling [64k,96k) all queue; all four freeze in D-state, survive kill -9 of writers and dd children, and REMAIN hung after an explicit 'vinum setstate -f down' revive abort (run.log). The queued struct requests and buffers leak until reboot. With a filesystem mounted on the volume, the victims are unprivileged writers. Fix (drain_sd_waitlist() hooked into set_sd_state() on every real transition out of sd_reviving: relaunch if the sd came up, biodone(EIO)+freerq otherwise) validated on the same guest: post-abort ps shows no hung writers (fix_run.log) vs stock's four immortal D-state dd's.
No comments yet.