Degraded RAID5 I/O: per-chunk ata_request leaked and bio never completed β boot wedge / unkillable hangs
Summary
In DEGRADED state the RAID5 branch of ata_raid_strategy falls through the empty 'do the XOR game' placeholder: the request allocated is neither queued nor freed and biodone() is never called. A crafted Promise/FreeBSD RAID5 metadata disk (width=2, total_disks=2, one member absent) assembled DEGRADED at boot and the kernel's own ar0 partition-scan bio hung forever - serial output stops exactly at the ar0 lines, ssh never came up, hard reset required (100% reproducible). The same hole hangs every read/write and leaks one ata_request per chunk on legitimately-built degraded RAID5 arrays. FIX VALIDATED on rebuilt kernel: array still announced DEGRADED, I/O errors cleanly, boot reaches login (v2 fix avoiding the CALLOUT_DID_INIT latent assert shared by pre-existing error paths).
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3696 Β· 12 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 2.1 KB | β raw | |
| VERDICT.md | β | 2.2 KB | β raw | |
| mkmeta.py | β | 4.0 KB | view raw | |
| build.sh | β | 133 B | view raw | |
| run.sh | β | 140 B | view raw | |
| run.log | β | 534 B | view raw | |
| panic.txt | β | 10.6 KB | view raw | |
| run.patched.log | β | 13.3 KB | view raw | |
| fix.diff | β | 3.8 KB | view raw | |
| env.txt | β | 505 B | view raw | |
| verdict.json | β | 2.4 KB | view raw | |
| manifest.json | β | 3.1 KB | view raw |
DF-3696 β Degraded RAID5 I/O: request leaked and bio never completed β wedge / unbootable system
Root cause (source)
ata_raid_strategy allocates an ata_request per chunk (:386), then for
RAID5 in DEGRADED state does nothing with it β the recovery path ("XOR
game") is unimplemented and the fall-through neither queues nor frees the
request nor completes the bio:
609: if (rdp->status & AR_S_DEGRADED) {
610: /* do the XOR game if possible */
611: }
612: else {
613: ... ata_raid_send_request(request); ...
}
break;
Consequences for every bio issued to a degraded RAID5 array:
- the ata_request allocated at :386 is leaked (objcache exhaustion /
kernel memory leak, one per chunk);
- bp->b_resid never reaches 0 and biodone(bp) is never called β the
caller sleeps forever in biowait (unkillable disk I/O).
Reproduce
f7_r5deg.img = FreeBSD/Promise RAID5 metadata: width=2, total_disks=2,
disks[0] ONLINE, disks[1] absent β array comes up DEGRADED (READY kept).
Boot with that single disk attached:
ar0: WARNING - parity protection lost. RAID5 array in DEGRADED mode ar0: 48MB <FreeBSD PseudoRAID RAID5 (stripe 0 KB)> status: DEGRADED ar0: disk0 READY using ad0 at ata0-master ar0: disk1 DOWN no device found for this subdisk
β¦and nothing else: the kernel's MBR slice-scan of ar0 blocks forever (note
the missing ar0: MBR magic not found line that a completing probe prints).
ssh never comes up; the system is unbootable while the disk is present.
Captured in panic.txt (serial console).
Expected
Baseline: boot wedges right after the ar0 lines (guest never reaches login;
vm.sh status β down).
Patched (fix.diff: degraded branch frees the request and fails the bio with
EIO): array announced, I/O errors out, boot proceeds to login
(run.patched.log).
Impact
Persistent local DoS from crafted metadata (system cannot boot); with an operator-built degraded RAID5, every read/write hangs its process and leaks kernel memory (CWE-401 + CWE-835).
Files
mkmeta.py (shared), run.sh, panic.txt, run.patched.log, fix.diff,
verdict.json
DF-3696 VERDICT
Status: reproduced (dos β boot-time system wedge; the "README expects hang/loop" case).
Reproduction
Crafted Promise/FreeBSD-native RAID5 metadata (width=2, total_disks=2, one
disk ONLINE, one absent) on ata0-master. The array assembled as DEGRADED
(READY retained by ata_raid_config_changed:955-965):
ar0: WARNING - parity protection lost. RAID5 array in DEGRADED mode ar0: 48MB <FreeBSD PseudoRAID RAID5 (stripe 0 KB)> status: DEGRADED ar0: disk0 READY using ad0 at ata0-master ar0: disk1 DOWN no device found for this subdisk
The kernel's slice-scan of /dev/ar0 then issued the first bio;
ata_raid_strategy:609-611 fell through the empty DEGRADED branch: the
request allocated at :386 was never queued or freed and the bio was never
completed. Serial output stops exactly there (no ar0: MBR magic not found
line, which requires a completed probe read); ssh never came up within 220 s;
vm.sh status β down. Guest hard-reset required.
Primitive characterization
- Demonstrated: full system wedge at boot (persistent DoS) from one crafted disk, zero interaction.
- Per-request kernel memory leak (
struct ata_requestfrom the objcache) for every chunk of every bio β source-certain (:386alloc vs no free/complete on the:609path); not separately metered in this run because the first hung bio already stops userspace from using the device. - The same hole affects runtime operation of legitimately-built degraded RAID5 arrays: each read/write hangs unkillably and leaks.
Fix validation
fix.diff replaces the empty branch with ata_free_request(request);
biofinish(bp, NULL, EIO); return(0); β matching the error style of the
neighboring paths. Patched kernel + same disk: ar0 announced DEGRADED, I/O
returns EIO, boot reaches login (run.patched.log). fix_status = fixed.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedv2 patched kernel boots with the same DEGRADED array, I/O errors cleanly, reaches login (see run.patched.log); v1 lesson documented in poc_changes
run.patched.log
Confirmed kernel references
Detail
Exploit chain
crafted disk -> DEGRADED ar0 at boot -> kernel slice-scan bio -> empty branch at :609-611 -> no biodone, request leaked -> boot wedge (persistent DoS); repeatable 100%
Evidence (decisive lines)
run.log/panic.txt (serial stops after the 3 ar0 lines; missing 'MBR magic' line proves the probe read never completed), run.patched.log (fixed kernel boots to login, I/O fails EIO)
PoC changes
first fix iteration used ata_free_request() on the never-queued request and tripped the CALLOUT_DID_INIT assert in callout_terminate (a latent flaw the pre-existing error paths at :423/:449/:624 share); corrected to bzero+objcache_put and re-validated
Verified recommended fix
fail the bio with EIO and free the un-queued request via bzero+objcache_put in the DEGRADED branch (fix.diff hunk 4, v2)
Verdict
In DEGRADED state the RAID5 branch of ata_raid_strategy falls through an empty 'XOR game' placeholder: the per-chunk ata_request allocated at :386 is neither queued nor freed and the bio is never completed. With a crafted Promise/FreeBSD RAID5 metadata disk (width=2, total_disks=2, one disk absent) the array assembled DEGRADED at boot and the kernel's own ar0 slice-scan bio hung forever - boot wedged before ssh, serial log stops exactly at the ar0 lines, hard reset required. Same hole hangs every read/write and leaks one ata_request per chunk on legitimately-built degraded RAID5 arrays.
No comments yet.