DF-2990 — UFS softdep newblk hash: lock-free insert vs locked remove

What

newblk_lookup() (sys/vfs/ufs/ffs_softdep.c:1003-1036) maintains the global newblk_hashtbl hash chains with no softdep lock (lk): both the lookup walk (newblk_find, :991) and the insertion (LIST_INSERT_HEAD, :1032) run lock-free. The two removal sites, softdep_setup_allocdirect() (:1317) and setup_allocindir_phase2() (:1633), remove entries from the same chains while holding lk (and kfree them at :1318/:1634). Every other hash in this file (pagedep :874/:1968/:3815, inodedep :966/:2122) is consistently protected by lk; the newblk table is the one structure with mixed discipline.

Attack surface / reachability

• insert: ffs_alloccg / ffs_alloccgfrag / ffs_alloccgblk → softdep_setup_blkmapdep (ffs_alloc.c:987, :1098, :1220) — runs on every block/frag allocation on a softdep UFS mount, i.e. on every unprivileged write(2) that extends a file.
• remove: ffs_balloc → softdep_setup_allocdirect (ffs_balloc.c:135, :188, :207, :268) / softdep_setup_allocindir_* — runs microseconds later in the same allocation sequence.

Two processes writing two different files on two CPUs both mutate the 65-chain table (only hashinit(64), :1062) concurrently with no common lock. The newblk_in_progress semaphore does not help: the remove path never takes it.

Failure modes

1. Interleaved insert/remove stores on one chain → lost head update → orphaned newblk → later consumer panics softdep_setup_allocdirect: lost block (:1302) / setup_allocindir: lost block (:1621).
2. Inserter's [oldfirst]->le_prev = &new->le_next fixup landing on an entry another CPU has already unlinked+freed (kfree at :1318) → use-after-free write into M_NEWBLK memory; symmetrically, a later LIST_REMOVE writing through a stale le_prev is a stray heap-pointer write into a live object (weakly controlled).
3. Lock-free newblk_find traversal stepping into an entry freed under lk → UAF read; after the freed slab is recycled (M_ZERO) the walk truncates early → false lost block panic.
4. On INVARIANTS kernels the corrupted chain is usually caught as Bad link elm %p next->prev != elm / Bad list head panics from queue.h QMD checks.

Amplifier

An I/O-completion interrupt preempting the inserter between its 2nd and 3rd store (between oldfirst->le_prev = &new->le_next and head->lh_first = new) stretches the effective race window from ~ns to the interrupt service time (~µs), during which any other CPU's locked remove of the old head completes the corruption deterministically. Hence heavy async metadata I/O + parallel allocation churn is the trigger.

Reproduce (in-guest)

# root: softdep UFS fs on vn0 (softdep is the newfs -U superblock flag;
# NOTE: mount -o softdep is NOT a DragonFly mount option)
dd if=/dev/zero of=/root/df2990.img bs=1m count=2048
vnconfig -c vn0 /root/df2990.img
newfs -U /dev/vn0
mount /dev/vn0 /mnt/df2990        # mount shows "(ufs, soft-updates)"
chmod 1777 /mnt/df2990
cc -O2 -o /tmp/df2990_churn churn.c
cc -O2 -o /tmp/df2990_frag frag_churn.c
/tmp/df2990_churn /mnt/df2990 900   # as any unprivileged user
/tmp/df2990_frag  /mnt/df2990 600   # frag-path variant

Success criterion: kernel panic (lost block / Bad link / found block) with the churn running, or orphaned-newblk corruption observable in dmesg. See VERDICT.md for the honest outcome of three bounded attempts on the 6-vCPU guest (~30 min saturated churn: no manifestation; the static defect is certain, the window is ns-scale).

Fix

Protect newblk_hashtbl with lk exactly like the pagedep/inodedep tables (see fix.diff).

DF-2990 VERDICT — newblk hash: lock-free insert vs locked remove

Finding

newblk_lookup() (sys/vfs/ufs/ffs_softdep.c:1003-1036) inserts into and walks the global newblk_hashtbl chains without the softdep lock lk, while softdep_setup_allocdirect() (:1317) and setup_allocindir_phase2() (:1633) LIST_REMOVE + kfree entries from the same 65 chains with lk held. pagedep (:874/:1968/:3815) and inodedep (:966/:2122) hashes are fully lk-protected; newblk is the only mixed-discipline table. Reachable by any unprivileged user doing parallel file-extension churn on a softdep UFS mount: insert side ffs_alloc.c:987/:1098/:1220 (softdep_setup_blkmapdep), remove side ffs_balloc.c:135/:188/:207/:268/:320/:376.

Static proof of the corrupting interleaving

LIST_INSERT_HEAD (sys/sys/queue.h:450) executes: (1) X->le_next = oldfirst; (2) oldfirst->le_prev = &X->le_next; (3) head->lh_first = X; (4) X->le_prev = &head->lh_first.

If the inserter is preempted (e.g. by an I/O-completion interrupt, which runs softdep_disk_write_complete and does not touch nb_hash) between (2) and (3), and another CPU then removes oldfirst under lk: LIST_REMOVE(oldfirst) writes *(oldfirst->le_prev) = oldfirst->le_next — but oldfirst->le_prev now points into X (store 2), so the remover's unlink lands in X->le_next and the head is left pointing at the about-to-be-freed oldfirst (kfree at :1318). Resumed store (3) then clobbers the head to X whose le_next was just rewritten. Net: chain head → freed memory / lost entries; subsequent locked removes write through stale le_prev pointers (stray heap-pointer store), and lookups traverse freed M_NEWBLK slabs (recycled via M_ZERO kmalloc at :1020) truncating walks → lost block panics (:1302/:1621). On INVARIANTS kernels (this guest: X86_64_GENERIC:56) the QMD list checks turn the first inconsistency into a Bad link elm panic.

Verification on the guest

Guest: DragonFly 6.5-DEVELOPMENT #0 X86_64_GENERIC (INVARIANTS), 6 vCPU KVM. UFS softdep image via vnconfig (newfs -U), mounted (ufs, soft-updates), churn executed as uid 1001 (unprivileged).

Attempt 1 (512MB fs, 10 procs, 600s): full-rate churn for ~5 min then softdep reclaim lag drove the fs to 100% ("filesystem full" console spam); no panic markers. Attempt 2 (2GB fs, bounded live set, parent sync() every 2s for reclaim + biodone interrupt storms, 900s): see run.2.log / run.log — outcome recorded below verbatim.

Outcome

not_reproduced (dynamically) — defect certain (statically). Three attempts on the 6-vCPU INVARIANTS guest, all as uid 1001 on a vn-backed newfs -U softdep mount:

1. 512MB fs, 10 procs, block churn, 600s — full-rate ~5 min, then softdep reclaim lag hit ENOSPC ("filesystem full" spam). No markers.
2. 2GB fs, 10 procs, block churn + parent sync()/2s, 900s — full-rate for the entire run (zero ENOSPC from these pids). No markers.
3. 4GB fs, 16 procs, frag-path churn (ffs_alloccg:1098 path) + sync()/1s, 600s — full-rate throughout. No markers.

≈30 minutes of saturated SMP allocation churn (~3 CPU-hours) with heavy biodone interrupt load; the softdep engine was deeply exercised (softdep_setup_freeblocks_bp teardown messages, rmdir/mkdir churn, ~500 in-flight newblks observed via vmstat -m). The corrupting window is the 3-4 store sequence of LIST_INSERT_HEAD (~ns), widened only when an interrupt lands exactly inside it and another CPU's locked remove of the same chain head completes during the preemption — historic equivalents of this race class need crash-atlas-scale run time, so non-reproduction in a bounded session does not refute the provable lock-discipline asymmetry. All kernel panics that WOULD result ("lost block", "Bad link") are unique to the race; none occur on this workload otherwise.

fix_status: not_testable — no firing trigger to validate against; the fix is a mechanical locking correction, verified git apply --check clean (fix.diff), mirroring pagedep/inodedep lookup discipline.

Fix

fix.diff: take lk around all nb_hash mutations in newblk_lookup (find-success, find-miss, allocation race-recheck, insert), with the kmalloc performed outside the lock — exactly the discipline already used by pagedep_lookup/inodedep_lookup. Also fixes the wrong- semaphore release on the race-avoided path (pagedep_in_progress → newblk_in_progress, the DF-0765 bug) as a side effect of using the interlock form.