β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-5071

amrd_strategy has no AMR_STATE_SHUTDOWN gate and nothing ever drains amr_bioq β€” bios park forever, issuers wedge in unkillable D-state (DF-2066/DF-5012/DF-5026 systemic class; distinct sink from DF-2082's in-flight UAF)

Summary

strategy lacks the gate its own amrd_open has (and even that gate is bypassed while another fd holds the disk); amr_startio swallows ENOMEM leaving bios queued, no watchdog exists, and shutdown/rescan/free never drain the queue - parked bios wedge issuers unkillably and at detach the queue head is kfree'd under them. Sink-level REPRODUCED on stock guest (D0/physstr survives kill -9, external-drain-only release); gate+drain 4-file fix validated (amr.ko -Werror, immediate ENXIO A/B).

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-5071 Β· 17 files
FileTypeDescriptionSize
df5071.c sink-module verbatim amrd_strategy/amr_submit_bio/amr_startio/amr_bio_command topology + verbatim amrd_open gate 11.2 KB view raw
df5071_fixed.c fixed-variant #define DF5071_FIXED wrapper (strategy gate + shutdown drain) 297 B view raw
df5071_trigger.c trigger shutdown/online/poke ioctls, wedge, wedgehold, freshopen 3.0 KB view raw
Makefile build 52 B ↓ download
Makefile.fixed build 59 B ↓ download
build.sh build-script 107 B view raw
build.log build-log final successful build (-Werror clean) 1.1 KB view raw
run.sh run-script baseline wedge demo 151 B view raw
run.log run-log D-state, kill -9 survival, poke release, gate asymmetry 2.2 KB view raw
run_ab.sh run-script fixed-variant A/B 1.3 KB view raw
run_ab.log run-log immediate ENXIO, no D-state, 4 bios drained at shutdown 993 B view raw
dmesg.txt kernel-log full df5071/df5071f dmesg trail 1.5 KB view raw
fix.diff suggested-fix amrd_strategy gate + amrd_fail_queued_bios() drains (4 files) 2.7 KB view raw
fix_validation.log fix-build-log patched amr.ko -Werror clean; sources restored 3.3 KB view raw
env.txt environment 502 B view raw
VERDICT.md verdict 5.0 KB ↓ raw
verdict.json verdict-machine 6.5 KB view raw
VERDICT.md verdict
↓ download raw

DF-5071 Verification Verdict

Severity: Medium Β· Impact class: dos (unkillable D-state hang, unbounded queue growth) Method: sink-level reproduction on the stock guest kernel (DF-4986/DF-5012/DF-5026 house precedent) + fix compile-validation against the guest's /usr/src.

Verdict: REPRODUCED (dos), fix VALIDATED (behavioral A/B + amr.ko build)

The bug (source chain, all lines verified)

Once AMR_STATE_SHUTDOWN is set (amr_pci_shutdown amr_pci.c:395 or amr_rescan_drives amr.c:466) β€” or the adapter simply wedges, since amr has no timeout/watchdog anywhere (verified: only a poll-loop timeout at amr.c:1355 exists, for polled commands) β€” every bio submitted through an already-open disk fd is parked on sc->amr_bioq forever:

  • amrd_strategy (amr_disk.c:166-194) checks ONLY sc == NULL; it has no AMR_STATE_SHUTDOWN gate β€” in the same file amrd_open (amr_disk.c:126-127) refuses new opens with ENXIO in exactly that state. Worse, even that lone gate is bypassed whenever another fd holds the disk open, because diskopen() skips the driver's d_open while a slice ref is open (sys/kern/subr_disk.c:1094) β€” demonstrated live.
  • amr_submit_bio (amr.c:407-416) enqueues (amrvar.h:309) and calls amr_startio, then returns 0.
  • amr_startio (amr.c:1155-1188) calls (void)amr_bio_command(sc, &ac) β€” the return value is IGNORED (amr.c:1171) β€” and gives up when no command was built (ac == NULL, amr.c:1178): the bio stays queued.
  • amr_bio_command (amr.c:1240-1241) returns ENOMEM without dequeuing when amr_alloccmd fails β€” the steady state on a wedged adapter where every command sits busy forever (nothing completes them: completions only come from the interrupt β†’ amr_done β†’ amr_complete β†’ amr_startio).
  • Nothing drains amr_bioq on any shutdown/detach path (amr_pci_shutdown amr_pci.c:387-417, amr_rescan_drives amr.c:454-485, amr_free amr.c:371-400); the comment at amr_pci.c:383 ("we can assume that the bioq on the controller is empty, as we won't allow shutdown if any device is open") is FALSE: AMR_STATE_OPEN is set only by the CONTROL device open (amr.c:430); amrd_open sets nothing. At detach the bioq head is kfreed with bios still on it (feeding the DF-2082 in-flight UAF window).

Consequence: the issuer sleeps in biowait in an unkillable D-state, and the queue grows without bound. Trigger: root kldunload amr / AMR_RESCAN_DRIVES ioctl with I/O queued or in flight, or any adapter wedge/fault. Root-on-amrd wedges the whole machine.

Reproduction (run.log)

  • holder process (fd opened BEFORE shutdown) pread after AMR_STATE_SHUTDOWN set β†’ D0/physstr (biowait);
  • survives kill -9 (SIGKILL pends; delivered only when the bio is eventually completed);
  • released only by the external poke drain (dmesg: poke -- draining 1 parked bio(s) with EIO), process exits with EIO;
  • amr_startup (amr.c:294-344, &= ~AMR_STATE_SHUTDOWN at :338) restores normal completion β€” parking is a direct consequence of the missing gate/drain, not of device removal;
  • gate asymmetry: fresh open during shutdown with holder β†’ SUCCEEDS (subr_disk.c:1094 bypass); fresh open without holder β†’ ENXIO (amrd_open :126-127 fires).

Fixed variant (run_ab.log)

Two hunks only (the ones in fix.diff): (1) amrd_strategy fail-fast mirroring amrd_open, (2) drain of queued bios when shutdown is entered. Result: shutdown entered with 4 incidental bios genuinely parked β†’ (fixed) draining 4 parked bio(s) with ENXIO β†’ the held fd's subsequent pread returns immediately r=-1 errno=6 (ENXIO) β€” no D-state, no residual process, no accumulation.

Fix validation

  1. Behavioral A/B at sink level (run.log vs run_ab.log): baseline wedges unkillably; fixed returns ENXIO immediately.
  2. Compile validation (fix_validation.log): fix.diff applied to the guest's /usr/src (amr_disk.c + amrvar.h + amr.c + amr_pci.c); make in /usr/src/sys/dev/raid/amr produced amr.ko with -Werror, 0 warnings (patched amr_disk.o in the link line). Sources restored pristine afterwards (grep marker 0).

Hardware gate

The real entry needs a MegaRAID adapter (absent on the QEMU guest); the completion topology is driver-side unconditional control flow, and the identical shape was sink-level reproduced for mfi (DF-5012) and ips (DF-5026) before. Impact ceiling: permanent unkillable-hang DoS for every issuer (whole-system wedge when amrd carries root) on hardware fault, adapter removal, or privileged rescan/unload.

Guest returned to a clean state (modules unloaded, sources pristine, no parked bios): guest_dirty = 0.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

fixed
baseline reproduced→ patch + rebuild →patched clean

Behavioral A/B at sink level (run.log vs run_ab.log): baseline pattern wedges unkillably (D-state survives SIGKILL, released only by external drain); fixed pattern (strategy fail-fast mirroring amrd_open + drain-on-shutdown) returns ENXIO immediately with no residual process and no accumulation (4 incidental bios drained at shutdown entry). fix.diff also applied to the guest's /usr/src (amr_disk.c, amrvar.h, amr.c, amr_pci.c) and compile-validated: full amr.ko rebuilt with -Werror clean including the patched amr_disk.o (fix_validation.log; sources restored pristine). The real amr entry is hardware-gated on this guest; the demonstrated failure mode and its removal are properties of the driver-side completion topology.

run_ab.log ('(fixed) draining 4 parked bio(s) with ENXIO' + 'wedgehold: pread@2M r=-1 errno=6' + 'wedgehold gone (completed)'); fix_validation.log (patched amr.ko -Werror clean); fix.diff
↓ fix.diffper-fix-DF-5071

Confirmed kernel references

Detail

Evidence (decisive lines)

run.log: '2237 D0 physstr df5071_trigger wedgehold' + survives 'kill -9' + 'poke -- draining 1 parked bio(s) with EIO' then process gone; dmesg: 'AMR_STATE_SHUTDOWN set; amr_bioq NOT drained (0 bio(s) already parked); amrd_strategy has NO shutdown gate'; gate asymmetry: freshopen-with-holder fd=3 ok (subr_disk.c:1094 bypass) vs freshopen-no-holder 'fd=-1 errno=6'; run_ab.log: '(fixed) draining 4 parked bio(s) with ENXIO' + 'wedgehold: pread@2M r=-1 errno=6' + 'wedgehold gone (completed)'

PoC changes

authored fresh from source analysis (no seed); three iterations were needed: (1) sim_startio originally looped forever on an empty queue (inverted havecmd semantics vs the real amr_startio break-on-ac==NULL) and wedged the guest -- fixed to the faithful translation; (2) control offsets initially sat at EOF for the declared media so toggles never reached strategy; (3) the parked counter originally counted submissions, not queued bios (re-ran both variants with truthful counts). Control plane lives on a separate ungated char device so the demo self-drives while amrd_open's verbatim gate stays on the raw disk.

Verified recommended fix

amrd_strategy: fail-fast (B_ERROR/ENXIO/b_resid/biodone) when amrd_controller->amr_state & AMR_STATE_SHUTDOWN, mirroring amrd_open; add amrd_fail_queued_bios() and call it from amr_pci_shutdown and amr_rescan_drives after AMR_STATE_SHUTDOWN is set (fix.diff, 4 files, amr.ko compile-validated).

Verdict

Sink-level reproduction on the stock guest kernel (DF-4986/DF-5012/DF-5026 house precedent): a raw D_DISK device wired to the verbatim amrd_strategy (amr_disk.c:166-194) -> amr_submit_bio (amr.c:407-416) -> amr_enqueue_bio (amrvar.h:309) -> amr_startio (amr.c:1155-1188, whose (void) cast at :1171 ignores amr_bio_command's ENOMEM from :1240-1241) topology demonstrates that once AMR_STATE_SHUTDOWN is set (amr_pci.c:395 / amr.c:466) or the adapter stops completing (amr has NO watchdog/timeout for async commands), every bio from an already-open fd parks on sc->amr_bioq forever: amrd_strategy has no SHUTDOWN gate while amrd_open in the SAME file refuses new opens (amr_disk.c:126-127), and even that lone gate is bypassed while another fd holds the disk because diskopen() skips the driver d_open (subr_disk.c:1094) -- both asymmetries shown live. The holder process landed in D0/physstr (biowait), SURVIVED kill -9 (SIGKILL pends), and was released only by an external drain (the poke) with EIO; amr_startup (:338) restored normal completion. Nothing in amr_pci_shutdown/amr_rescan_drives/amr_free ever drains amr_bioq -- the amr_pci.c:383 comment claiming the bioq must be empty is false since only the control device sets AMR_STATE_OPEN (amr.c:430); at detach the queue head itself is kfree'd under the parked bios (feeding the DF-2082 in-flight UAF window). With the fix pattern transplanted into the sink (strategy fail-fast mirroring amrd_open + drain on shutdown entry) the same held-fd read returns immediately r=-1 errno=6 (ENXIO): no D-state, no accumulation. The real amr entry is hardware-gated on this guest (no MegaRAID adapter) but the completion topology is driver-side unconditional control flow; the identical shape was previously sink-reproduced for mfi (DF-5012) and ips (DF-5026). Impact ceiling: permanent unkillable-hang DoS for every issuer (whole-system wedge when amrd carries root) on adapter fault/removal or privileged rescan/unload.