DF-5787 evidence pack — stale b_bio[12].bio_flags survive buffer-cache recycling

Finding (short)

sys/kern/vfs_bio.c never clears a buffer header's embedded b_bio1.bio_flags when the buffer is released and later reacquired:

• getnewbuf() resets bp->b_flags = B_BNOCLIP only (sys/kern/vfs_bio.c:2095) and calls reinitbufbio(), which resets bio_done/bio_offset but not bio_flags (sys/kern/vfs_bio.c:743-751).
• getblk()'s found-in-cache path performs no bio-flag hygiene either (sys/kern/vfs_bio.c:2746-2988).

The kernel itself documents that stale BIO_DONE is load-bearing: vn_strategy() has KKASSERT((bio->bio_flags & BIO_DONE) == 0) (sys/kern/vfs_bio.c:3385). Every in-tree completion path must clear the flags by hand (_biowait() clears BIO_DONE|BIO_SYNC on success at :3284; the breadcb/cluster_readcb callback contract puts clearing on the callback — see hammer's hammer_indirect_callback, hammer_io.c:1640). Any forgetful or abandoning completion path (callback bug, biowait_timeout abandonment) permanently poisons the buffer header, and the next bread()/bwrite() on that buffer:

1. panics at vn_strategy (:3385) on an always-KKASSERT kernel, or
2. on the biowait side, _biowait() observes the stale BIO_DONE and returns before the device I/O completes — the caller reads b_data while the device is still DMAing into it (torn data), then brelse()s the buffer with the I/O in flight, so the asynchronous biodone() later runs bpdone() on a requeued (possibly reused) buffer → queue corruption / double-accounting.

Rig

KLD module biopoison.c (root-only, simulates the documented-but-undefended caller-contract violation):

• Case 0 (control): getblk → clear B_CACHE → bqrelse → bread — must return 0 with no panic (proves rig mechanics).
• Case 1 (poison): getblk → clear B_CACHE and leave BIO_DONE set (exactly what breadcb()'s cached fast path does at vfs_bio.c:883, minus the callback's cleanup) → bqrelse → bread.

Result (baseline, unpatched kernel)

biopoison: vp=0xfffff80116870500 v_type=1            (hammer2 file vp, /etc/fstab)
biopoison: === control run (no stale flag) ===
biopoison[0]: getblk -> bp=0xfffff8004f7442d0 b_flags=21000160 bio1.flags=00000000 B_CACHE=1
biopoison[0]: released (stale BIO_DONE=0), issuing bread()
biopoison[0]: bread returned 0 (no panic)
biopoison: === poisoned run (stale BIO_DONE across bqrelse -> getblk recycling) ===
biopoison[1]: getblk -> bp=0xfffff8004f7442d0 b_flags=21001164 bio1.flags=00000000 B_CACHE=1
biopoison[1]: released (stale BIO_DONE=1), issuing bread()
panic: assertion "(bio->bio_flags & BIO_DONE) == 0" failed in vn_strategy at /usr/src/sys/kern/vfs_bio.c:3385
Trace: vn_strategy <- breadnx <- bpio_case <- biopoison_handler <- module_register_init

Same buffer header (0xfffff8004f7442d0) in both cases — the flag crossed the bqrelse → queue → getblk boundary. Verdict: reproduced (panic; guest entered ddb, ssh session died).

Build

# on the guest (kernel obj tree must exist):
mkdir -p /root/biopoison && cd /root/biopoison    # biopoison.c + Makefile here
make                                             # SYSDIR=/usr/src/sys in Makefile
# -> biopoison.ko

Run

kldload /root/biopoison/biopoison.ko     # console/serial shows the output above

Fix validation

fix.diff: assignment-style flag arming in breadnx()/bwrite() (bp->b_bio1.bio_flags = BIO_SYNC; instead of |=), plus reinitbufbio() clearing bio_flags for recycled headers. After building the patched kernel in the guest (make quickkernel && make installkernel, reboot), the identical module load runs both cases with no panic — see VERDICT.md.

DF-5787 — VERDICT

Status: reproduced (kernel panic via KLD rig; fix validated fixed)

What was audited and claimed

Pass-3 audit of sys/kern/vfs_bio.c (4659 LOC) across the five mandated hunt areas (buffer lifecycle/recycling races, biodone/brelse + B_DONE discipline, KVA qenter/qremove pairing, bufdaemon flush logic, unmount/fsync races). Beyond the excluded known finding DF-2675 (allocbuf 17-page KVA overflow), one new defect was found and verified:

Buffer-cache recycling performs zero hygiene on the embedded BIO completion flags. A buffer header released with b_bio1.bio_flags bits set (BIO_DONE/BIO_WANT/BIO_SYNC) carries them into its next use:

• getnewbuf() resets only bp->b_flags = B_BNOCLIP — sys/kern/vfs_bio.c:2095
• reinitbufbio() resets bio_done/bio_offset only — sys/kern/vfs_bio.c:747-750
• getblk()'s found-in-cache path does not touch bio_flags at all — sys/kern/vfs_bio.c:2746-2988
• breadnx()/bwrite() arm the sync-completion bits with |=, preserving any stale BIO_DONE — sys/kern/vfs_bio.c:919-920, 989-990

The kernel itself treats stale BIO_DONE as an invalid state: KKASSERT((bio->bio_flags & BIO_DONE) == 0) in vn_strategy() — sys/kern/vfs_bio.c:3385 (KKASSERT is unconditional in DragonFly). The only reason this does not fire in-tree is that every completion path carefully clears the flags by hand (_biowait() at :3284 on success; the breadcb()/cluster_readcb() contract puts clearing on the callback — see hammer_indirect_callback(), sys/vfs/hammer/hammer_io.c:1640 "bio->bio_flags &= ~(BIO_DONE | BIO_SYNC)"). The breadcb() cached-path itself sets BIO_DONE and releases (sys/kern/vfs_bio.c:883) — one forgetful callback, or a biowait_timeout() abandonment, permanently poisons the buffer header.

Reproduction (baseline, kernel #0 Thu Jul 2, unpatched)

KLD biopoison.c (root-only rig; emulates the documented-but-undefended contract violation — the same state breadcb()'s fast path leaves behind, minus the callback's cleanup):

biopoison[0]: getblk -> bp=0xfffff8004f7442d0 ... B_CACHE=1
biopoison[0]: released (stale BIO_DONE=0), issuing bread()
biopoison[0]: bread returned 0 (no panic)              <- control passes
biopoison[1]: getblk -> bp=0xfffff8004f7442d0 ...      <- SAME header, found path
biopoison[1]: released (stale BIO_DONE=1), issuing bread()
panic: assertion "(bio->bio_flags & BIO_DONE) == 0" failed in vn_strategy at /usr/src/sys/kern/vfs_bio.c:3385
Trace: vn_strategy <- breadnx <- bpio_case <- biopoison_handler <- module_register_init

Full serial output: run.log / panic.txt. Guest entered ddb; ssh died; vm.sh status → down-equivalent (DDB). Same buffer address in both cases proves the flag crossed bqrelse() → clean queue → getblk() reacquisition.

On a kernel without the (always-on) KKASSERT the next failure point is worse: _biowait() (sys/kern/vfs_bio.c:3261-3263) observes the stale BIO_DONE and returns before the device I/O completes — the caller consumes b_data while the device is still DMAing into the pages (torn data), then brelse()s the buffer with I/O in flight, so the asynchronous biodone() → bpdone() later runs on a requeued (possibly reused) buffer: queue corruption, double-accounting, or the _biowait KKASSERT at :3283 (KKASSERT(bp->b_cmd == BUF_CMD_DONE)).

Threat model

This is a kernel-internal robustness / defense-in-depth defect, not an unprivileged attack primitive: the poison requires a kernel-side completion path that releases a bio with BIO_DONE set (buggy out-of-tree or future VFS/driver callback, or biowait_timeout() abandonment as in sys/dev/disk/fd/fd.c:2279). No in-tree trigger exists today; every current caller honors the contract. Severity Low.

Fix validation (fix.diff, kernel #1 Mon Oct 12 16:00:55)

fix.diff (3 hunks, applies cleanly to sys/): 1. reinitbufbio(): clear bio->bio_flags (covers recycled headers via getnewbuf() :2104 and kern_physio's re-arm, sys/kern/kern_physio.c:68). 2. breadnx() :921: bp->b_bio1.bio_flags = BIO_SYNC; (assignment, not |=). 3. bwrite() :991: same assignment arming.

Procedure: vm.sh reset with-src → patch /usr/src → make -j4 quickkernel (vfs_bio.o rebuilt, 55 s) → make installkernel → reboot → rebuild the identical module → kldload:

biopoison[0]: bread returned 0 (no panic)              <- control
biopoison[1]: released (stale BIO_DONE=1), issuing bread()
biopoison[1]: bread returned 0 (no panic)              <- panic GONE
biopoison: bread SURVIVED - stale flag was cleared somewhere during recycling

Guest stayed up (kldstat shows biopoison.ko loaded, uptime continues). Baseline panic vs patched no-panic: fixed. Guest reset to with-src afterwards; no /usr/src modifications remain in the shipped tree (sys/ is untouched — fix.diff was only applied inside the disposable guest).

Negative results (classes hunted and killed in pass 3)

• getnewbuf/findblk recycling race: b_refs is checked unsynchronized then re-checked under lock+spin (sys/kern/vfs_bio.c:2002,2031,2131); brelvp() under v_token closes the findblk 0→1 window. No hole found.
• brelse's manual B_DELWRI unwind (:1311-1320): unreachable — buf_deallocate callbacks (hammer_io_deallocate) never set B_DELWRI.
• KVA qenter/qremove pairing: every qenter/qremove is symmetric at b_kvabase with bkvareset()/CPUMASK_ASSZERO on each mapping change (:1444-1451, :1854-1856, :3077-3094, :3212-3216, :3694-3698, :3834-3838, :3991-3994). The only asymmetry is the known DF-2675 (17-page mapping).
• brelse page-invalidation loop: resid can go negative only after the final iteration; presid never sees it (KASSERT safe) — killed.
• bufdaemon flush queue protocol: all queue mutations under pcpu spin with buffer locked; marker discipline correct; B_DEFERRED livelock bounded.
• bio_track_rel underflow: guarded by unconditional bit-30 panic (:548-549).
• getnewbuf()'s size/maxsize parameters are dead (nothing enforces the misaligned-loffset headroom getblk computes at :2939) — cosmetic, since every header owns a full MAXBSIZE slot; only DF-2675 defeats that.
• inmem() would spin forever if f_iosize == 0 (:2503-2522, tinc == 0) — no in-tree mount produces iosize 0; noted, not filed.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.