# DF-2990 VERDICT — newblk hash: lock-free insert vs locked remove

## Finding

`newblk_lookup()` (sys/vfs/ufs/ffs_softdep.c:1003-1036) inserts into and
walks the global `newblk_hashtbl` chains **without** the softdep lock
`lk`, while `softdep_setup_allocdirect()` (:1317) and
`setup_allocindir_phase2()` (:1633) `LIST_REMOVE` + `kfree` entries from
the same 65 chains **with** `lk` held. pagedep (:874/:1968/:3815) and
inodedep (:966/:2122) hashes are fully `lk`-protected; newblk is the
only mixed-discipline table. Reachable by any unprivileged user doing
parallel file-extension churn on a softdep UFS mount:
insert side ffs_alloc.c:987/:1098/:1220 (`softdep_setup_blkmapdep`),
remove side ffs_balloc.c:135/:188/:207/:268/:320/:376.

## Static proof of the corrupting interleaving

`LIST_INSERT_HEAD` (sys/sys/queue.h:450) executes:
(1) `X->le_next = oldfirst`; (2) `oldfirst->le_prev = &X->le_next`;
(3) `head->lh_first = X`; (4) `X->le_prev = &head->lh_first`.

If the inserter is preempted (e.g. by an I/O-completion interrupt, which
runs `softdep_disk_write_complete` and does not touch nb_hash) between
(2) and (3), and another CPU then removes `oldfirst` under `lk`:
`LIST_REMOVE(oldfirst)` writes `*(oldfirst->le_prev) = oldfirst->le_next`
— but `oldfirst->le_prev` now points **into X** (store 2), so the
remover's unlink lands in `X->le_next` and the head is left pointing at
the about-to-be-freed `oldfirst` (kfree at :1318). Resumed store (3)
then clobbers the head to X whose `le_next` was just rewritten. Net:
chain head → freed memory / lost entries; subsequent locked removes
write through stale `le_prev` pointers (stray heap-pointer store), and
lookups traverse freed M_NEWBLK slabs (recycled via M_ZERO kmalloc at
:1020) truncating walks → `lost block` panics (:1302/:1621). On
INVARIANTS kernels (this guest: X86_64_GENERIC:56) the QMD list checks
turn the first inconsistency into a `Bad link elm` panic.

## Verification on the guest

Guest: DragonFly 6.5-DEVELOPMENT #0 X86_64_GENERIC (INVARIANTS), 6 vCPU
KVM. UFS softdep image via vnconfig (`newfs -U`), mounted
`(ufs, soft-updates)`, churn executed as uid 1001 (unprivileged).

Attempt 1 (512MB fs, 10 procs, 600s): full-rate churn for ~5 min then
softdep reclaim lag drove the fs to 100% ("filesystem full" console
spam); no panic markers. Attempt 2 (2GB fs, bounded live set, parent
sync() every 2s for reclaim + biodone interrupt storms, 900s): see
run.2.log / run.log — outcome recorded below verbatim.

## Outcome

**not_reproduced (dynamically) — defect certain (statically).** Three
attempts on the 6-vCPU INVARIANTS guest, all as uid 1001 on a
vn-backed `newfs -U` softdep mount:

1. 512MB fs, 10 procs, block churn, 600s — full-rate ~5 min, then
   softdep reclaim lag hit ENOSPC ("filesystem full" spam). No markers.
2. 2GB fs, 10 procs, block churn + parent sync()/2s, 900s — full-rate
   for the entire run (zero ENOSPC from these pids). No markers.
3. 4GB fs, 16 procs, frag-path churn (ffs_alloccg:1098 path) +
   sync()/1s, 600s — full-rate throughout. No markers.

≈30 minutes of saturated SMP allocation churn (~3 CPU-hours) with heavy
biodone interrupt load; the softdep engine was deeply exercised
(`softdep_setup_freeblocks_bp` teardown messages, rmdir/mkdir churn,
~500 in-flight newblks observed via vmstat -m). The corrupting window
is the 3-4 store sequence of LIST_INSERT_HEAD (~ns), widened only when
an interrupt lands exactly inside it and another CPU's locked remove of
the same chain head completes during the preemption — historic
equivalents of this race class need crash-atlas-scale run time, so
non-reproduction in a bounded session does not refute the provable
lock-discipline asymmetry. All kernel panics that WOULD result
("lost block", "Bad link") are unique to the race; none occur on this
workload otherwise.

fix_status: not_testable — no firing trigger to validate against; the
fix is a mechanical locking correction, verified `git apply --check`
clean (fix.diff), mirroring pagedep/inodedep lookup discipline.

## Fix

fix.diff: take `lk` around all nb_hash mutations in `newblk_lookup`
(find-success, find-miss, allocation race-recheck, insert), with the
kmalloc performed outside the lock — exactly the discipline already
used by `pagedep_lookup`/`inodedep_lookup`. Also fixes the wrong-
semaphore release on the race-avoided path (pagedep_in_progress →
newblk_in_progress, the DF-0765 bug) as a side effect of using the
interlock form.