amrd_strategy has no AMR_STATE_SHUTDOWN gate and nothing ever drains amr_bioq β bios park forever, issuers wedge in unkillable D-state (DF-2066/DF-5012/DF-5026 systemic class; distinct sink from DF-2082's in-flight UAF)
Summary
strategy lacks the gate its own amrd_open has (and even that gate is bypassed while another fd holds the disk); amr_startio swallows ENOMEM leaving bios queued, no watchdog exists, and shutdown/rescan/free never drain the queue - parked bios wedge issuers unkillably and at detach the queue head is kfree'd under them. Sink-level REPRODUCED on stock guest (D0/physstr survives kill -9, external-drain-only release); gate+drain 4-file fix validated (amr.ko -Werror, immediate ENXIO A/B).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-5071 Β· 17 files| File | Type | Description | Size | |
|---|---|---|---|---|
| df5071.c | sink-module | verbatim amrd_strategy/amr_submit_bio/amr_startio/amr_bio_command topology + verbatim amrd_open gate | 11.2 KB | view raw |
| df5071_fixed.c | fixed-variant | #define DF5071_FIXED wrapper (strategy gate + shutdown drain) | 297 B | view raw |
| df5071_trigger.c | trigger | shutdown/online/poke ioctls, wedge, wedgehold, freshopen | 3.0 KB | view raw |
| Makefile | build | 52 B | β download | |
| Makefile.fixed | build | 59 B | β download | |
| build.sh | build-script | 107 B | view raw | |
| build.log | build-log | final successful build (-Werror clean) | 1.1 KB | view raw |
| run.sh | run-script | baseline wedge demo | 151 B | view raw |
| run.log | run-log | D-state, kill -9 survival, poke release, gate asymmetry | 2.2 KB | view raw |
| run_ab.sh | run-script | fixed-variant A/B | 1.3 KB | view raw |
| run_ab.log | run-log | immediate ENXIO, no D-state, 4 bios drained at shutdown | 993 B | view raw |
| dmesg.txt | kernel-log | full df5071/df5071f dmesg trail | 1.5 KB | view raw |
| fix.diff | suggested-fix | amrd_strategy gate + amrd_fail_queued_bios() drains (4 files) | 2.7 KB | view raw |
| fix_validation.log | fix-build-log | patched amr.ko -Werror clean; sources restored | 3.3 KB | view raw |
| env.txt | environment | 502 B | view raw | |
| VERDICT.md | verdict | 5.0 KB | β raw | |
| verdict.json | verdict-machine | 6.5 KB | view raw |
DF-5071 Verification Verdict
Severity: Medium Β· Impact class: dos (unkillable D-state hang, unbounded queue growth) Method: sink-level reproduction on the stock guest kernel (DF-4986/DF-5012/DF-5026 house precedent) + fix compile-validation against the guest's /usr/src.
Verdict: REPRODUCED (dos), fix VALIDATED (behavioral A/B + amr.ko build)
The bug (source chain, all lines verified)
Once AMR_STATE_SHUTDOWN is set (amr_pci_shutdown amr_pci.c:395 or
amr_rescan_drives amr.c:466) β or the adapter simply wedges, since amr
has no timeout/watchdog anywhere (verified: only a poll-loop timeout
at amr.c:1355 exists, for polled commands) β every bio submitted through
an already-open disk fd is parked on sc->amr_bioq forever:
amrd_strategy(amr_disk.c:166-194) checks ONLYsc == NULL; it has noAMR_STATE_SHUTDOWNgate β in the same fileamrd_open(amr_disk.c:126-127) refuses new opens with ENXIO in exactly that state. Worse, even that lone gate is bypassed whenever another fd holds the disk open, becausediskopen()skips the driver'sd_openwhile a slice ref is open (sys/kern/subr_disk.c:1094) β demonstrated live.amr_submit_bio(amr.c:407-416) enqueues (amrvar.h:309) and callsamr_startio, then returns 0.amr_startio(amr.c:1155-1188) calls(void)amr_bio_command(sc, &ac)β the return value is IGNORED (amr.c:1171) β and gives up when no command was built (ac == NULL, amr.c:1178): the bio stays queued.amr_bio_command(amr.c:1240-1241) returns ENOMEM without dequeuing whenamr_alloccmdfails β the steady state on a wedged adapter where every command sits busy forever (nothing completes them: completions only come from the interrupt βamr_doneβamr_completeβamr_startio).- Nothing drains
amr_bioqon any shutdown/detach path (amr_pci_shutdownamr_pci.c:387-417,amr_rescan_drivesamr.c:454-485,amr_freeamr.c:371-400); the comment at amr_pci.c:383 ("we can assume that the bioq on the controller is empty, as we won't allow shutdown if any device is open") is FALSE:AMR_STATE_OPENis set only by the CONTROL device open (amr.c:430);amrd_opensets nothing. At detach the bioq head iskfreed with bios still on it (feeding the DF-2082 in-flight UAF window).
Consequence: the issuer sleeps in biowait in an unkillable D-state,
and the queue grows without bound. Trigger: root kldunload amr /
AMR_RESCAN_DRIVES ioctl with I/O queued or in flight, or any adapter
wedge/fault. Root-on-amrd wedges the whole machine.
Reproduction (run.log)
- holder process (fd opened BEFORE shutdown) pread after
AMR_STATE_SHUTDOWNset βD0/physstr(biowait); - survives
kill -9(SIGKILL pends; delivered only when the bio is eventually completed); - released only by the external poke drain (
dmesg: poke -- draining 1 parked bio(s) with EIO), process exits with EIO; amr_startup(amr.c:294-344,&= ~AMR_STATE_SHUTDOWNat :338) restores normal completion β parking is a direct consequence of the missing gate/drain, not of device removal;- gate asymmetry: fresh open during shutdown with holder β SUCCEEDS (subr_disk.c:1094 bypass); fresh open without holder β ENXIO (amrd_open :126-127 fires).
Fixed variant (run_ab.log)
Two hunks only (the ones in fix.diff): (1) amrd_strategy fail-fast
mirroring amrd_open, (2) drain of queued bios when shutdown is entered.
Result: shutdown entered with 4 incidental bios genuinely parked β
(fixed) draining 4 parked bio(s) with ENXIO β the held fd's subsequent
pread returns immediately r=-1 errno=6 (ENXIO) β no D-state, no
residual process, no accumulation.
Fix validation
- Behavioral A/B at sink level (run.log vs run_ab.log): baseline wedges unkillably; fixed returns ENXIO immediately.
- Compile validation (fix_validation.log):
fix.diffapplied to the guest's/usr/src(amr_disk.c + amrvar.h + amr.c + amr_pci.c);makein/usr/src/sys/dev/raid/amrproducedamr.kowith-Werror, 0 warnings (patchedamr_disk.oin the link line). Sources restored pristine afterwards (grep marker 0).
Hardware gate
The real entry needs a MegaRAID adapter (absent on the QEMU guest); the completion topology is driver-side unconditional control flow, and the identical shape was sink-level reproduced for mfi (DF-5012) and ips (DF-5026) before. Impact ceiling: permanent unkillable-hang DoS for every issuer (whole-system wedge when amrd carries root) on hardware fault, adapter removal, or privileged rescan/unload.
Guest returned to a clean state (modules unloaded, sources pristine,
no parked bios): guest_dirty = 0.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedBehavioral A/B at sink level (run.log vs run_ab.log): baseline pattern wedges unkillably (D-state survives SIGKILL, released only by external drain); fixed pattern (strategy fail-fast mirroring amrd_open + drain-on-shutdown) returns ENXIO immediately with no residual process and no accumulation (4 incidental bios drained at shutdown entry). fix.diff also applied to the guest's /usr/src (amr_disk.c, amrvar.h, amr.c, amr_pci.c) and compile-validated: full amr.ko rebuilt with -Werror clean including the patched amr_disk.o (fix_validation.log; sources restored pristine). The real amr entry is hardware-gated on this guest; the demonstrated failure mode and its removal are properties of the driver-side completion topology.
run_ab.log ('(fixed) draining 4 parked bio(s) with ENXIO' + 'wedgehold: pread@2M r=-1 errno=6' + 'wedgehold gone (completed)'); fix_validation.log (patched amr.ko -Werror clean); fix.diff
Confirmed kernel references
- sys/dev/raid/amr/amr_disk.c:166
- sys/dev/raid/amr/amr_disk.c:175
- sys/dev/raid/amr/amr_disk.c:126
- sys/dev/raid/amr/amr_disk.c:181
- sys/dev/raid/amr/amr.c:407
- sys/dev/raid/amr/amr.c:411
- sys/dev/raid/amr/amr.c:1155
- sys/dev/raid/amr/amr.c:1171
- sys/dev/raid/amr/amr.c:1178
- sys/dev/raid/amr/amr.c:1240
- sys/dev/raid/amr/amrvar.h:309
- sys/dev/raid/amr/amr_pci.c:383
- sys/dev/raid/amr/amr_pci.c:394
- sys/dev/raid/amr/amr.c:430
- sys/dev/raid/amr/amr.c:466
- sys/kern/subr_disk.c:1094
Detail
Evidence (decisive lines)
run.log: '2237 D0 physstr df5071_trigger wedgehold' + survives 'kill -9' + 'poke -- draining 1 parked bio(s) with EIO' then process gone; dmesg: 'AMR_STATE_SHUTDOWN set; amr_bioq NOT drained (0 bio(s) already parked); amrd_strategy has NO shutdown gate'; gate asymmetry: freshopen-with-holder fd=3 ok (subr_disk.c:1094 bypass) vs freshopen-no-holder 'fd=-1 errno=6'; run_ab.log: '(fixed) draining 4 parked bio(s) with ENXIO' + 'wedgehold: pread@2M r=-1 errno=6' + 'wedgehold gone (completed)'
PoC changes
authored fresh from source analysis (no seed); three iterations were needed: (1) sim_startio originally looped forever on an empty queue (inverted havecmd semantics vs the real amr_startio break-on-ac==NULL) and wedged the guest -- fixed to the faithful translation; (2) control offsets initially sat at EOF for the declared media so toggles never reached strategy; (3) the parked counter originally counted submissions, not queued bios (re-ran both variants with truthful counts). Control plane lives on a separate ungated char device so the demo self-drives while amrd_open's verbatim gate stays on the raw disk.
Verified recommended fix
amrd_strategy: fail-fast (B_ERROR/ENXIO/b_resid/biodone) when amrd_controller->amr_state & AMR_STATE_SHUTDOWN, mirroring amrd_open; add amrd_fail_queued_bios() and call it from amr_pci_shutdown and amr_rescan_drives after AMR_STATE_SHUTDOWN is set (fix.diff, 4 files, amr.ko compile-validated).
Verdict
Sink-level reproduction on the stock guest kernel (DF-4986/DF-5012/DF-5026 house precedent): a raw D_DISK device wired to the verbatim amrd_strategy (amr_disk.c:166-194) -> amr_submit_bio (amr.c:407-416) -> amr_enqueue_bio (amrvar.h:309) -> amr_startio (amr.c:1155-1188, whose (void) cast at :1171 ignores amr_bio_command's ENOMEM from :1240-1241) topology demonstrates that once AMR_STATE_SHUTDOWN is set (amr_pci.c:395 / amr.c:466) or the adapter stops completing (amr has NO watchdog/timeout for async commands), every bio from an already-open fd parks on sc->amr_bioq forever: amrd_strategy has no SHUTDOWN gate while amrd_open in the SAME file refuses new opens (amr_disk.c:126-127), and even that lone gate is bypassed while another fd holds the disk because diskopen() skips the driver d_open (subr_disk.c:1094) -- both asymmetries shown live. The holder process landed in D0/physstr (biowait), SURVIVED kill -9 (SIGKILL pends), and was released only by an external drain (the poke) with EIO; amr_startup (:338) restored normal completion. Nothing in amr_pci_shutdown/amr_rescan_drives/amr_free ever drains amr_bioq -- the amr_pci.c:383 comment claiming the bioq must be empty is false since only the control device sets AMR_STATE_OPEN (amr.c:430); at detach the queue head itself is kfree'd under the parked bios (feeding the DF-2082 in-flight UAF window). With the fix pattern transplanted into the sink (strategy fail-fast mirroring amrd_open + drain on shutdown entry) the same held-fd read returns immediately r=-1 errno=6 (ENXIO): no D-state, no accumulation. The real amr entry is hardware-gated on this guest (no MegaRAID adapter) but the completion topology is driver-side unconditional control flow; the identical shape was previously sink-reproduced for mfi (DF-5012) and ips (DF-5026). Impact ceiling: permanent unkillable-hang DoS for every issuer (whole-system wedge when amrd carries root) on adapter fault/removal or privileged rescan/unload.
No comments yet.