DF-2645 — flush-driven indirect-block collapse racing a concurrent child modification

Class: lock/refcount race in the flusher's indirect-maintenance path, reached from hammer2_flush_core() (sys/vfs/hammer2/hammer2_flush.c:1039-1042).

Reach: unprivileged local user (file create/delete + append + sync on a mounted hammer2 PFS).

Status: not_reproduced after 2 stress attempts (~35 min, >130M appends + continuous churn + 20 Hz sync loop); race is code-proven (confidence: likely). See VERDICT.md for the full analysis and run.log for the experiment transcripts.

Reproduce

cc -O2 -o stress2645 stress2645.c
cc -O2 -o mknames   mknames.c
cc -O2 -o mkfiles   mkfiles.c
cc -O2 -o delsome   delsome.c
sh run_df2645.sh 12 1        # 512MB vn image, 6000-file dir, 70% delete,
                             # 4 append writers + create/delete churn +
                             # sync loop

Success criterion (not achieved): kernel panic with the hammer2_chain_repchange KKASSERT (hammer2_chain.c:2314) or hammer2: debug repchange console prints; alternatively any LOST CHILD/overlap marker from the maintenance path.

Mechanism (why it should be possible)

1. flush_core(B) [B = sparse INDIRECT with UPDATE set] scans+flushes B's children bottom-up, updating B's media blocktable per child (hammer2_flush.c:1116-1134).
2. A frontend write modifies a file-inode chain F under B AFTER F's flush completes but BEFORE flush_core(B) reaches hammer2_chain_indirect_maintenance(parent, B) (:1040) — hammer2_chain_modify() COWs F to a fresh data_off (hammer2_chain.c:1437+), so F's in-memory bref diverges from B's media entry.
3. maintenance's collapse loop hits the skip guard (bcmp(&bsave, &sub->bref) mismatch, chain.c:4202-4209) and leaves F in B's rbtree.
4. hammer2_chain_repchange(parent, B) (chain.c:4228 → :2314) unconditionally KKASSERTs B is empty → panic (DoS); on non-INVARIANTS semantics the reptracks migrate while F is still parented to the already-deleted B → F's media entry becomes unreachable (silent data loss).

Suggested fix

Defer the destructive hammer2_chain_delete(parent, chain) (chain.c:4160) until after a clean full sweep of the collapse loop; abort the collapse (return 0, flush retries later) when any child was skipped, keeping repchange's invariant true.

DF-2645 — flush-driven indirect collapse vs concurrent child modify → repchange KKASSERT / orphaned subtree

Status: not_reproduced (2 stress attempts, ~35 min of racing; race is code-proven but was not won; attempt 2 degraded into an ENOSPC flush storm — DF-2633 family behavior — before the window could be hit).

Root-cause analysis (code-proven)

hammer2_flush_core() calls hammer2_chain_indirect_maintenance(parent, chain) for every INDIRECT child with UPDATE set (sys/vfs/hammer2/hammer2_flush.c:1039-1042). The collapse loop in maintenance knows about concurrent modification races and skips children whose in-memory bref no longer matches the parent's media entry:

        if (bcmp(&bsave, &sub->bref) ||          /* chain.c:4202 */
            sub->parent != chain ||
            (sub->flags & HAMMER2_CHAIN_DELETED)) {
            hammer2_chain_unlock(sub);       /* skip, not moved */
            ...
            continue;
        }

…but the code after the loop assumes the loop emptied the chain:

    hammer2_chain_repchange(parent, chain);          /* chain.c:4228 */

hammer2_chain_repchange(...)
{
    KKASSERT(chain->core.live_count == 0 && RB_EMPTY(&chain->core.rbtree));
                                /* chain.c:2314 — UNCONDITIONAL KKASSERT */

The two are mutually inconsistent under exactly the race the skip guard anticipates: a child that was flushed by this pass's downward scan (flush_recurse, hammer2_flush.c:1163-1257) and then modified again by the frontend before the bottom-up phase reaches maintenance has a fresh COW bref (new data_off from hammer2_chain_modify, chain.c:1437+) that no longer matches the parent's media blockref; the skip guard leaves it in the rbtree with live_count ≥ 1; repchange's KKASSERT fires → panic on INVARIANTS kernels (DragonFly KKASSERT is unconditional) and mis-migrated reptracks + silently orphaned subtree on others (the chain's own bref was already deleted from the grandparent at chain.c:4160, so the skipped child's media entry becomes unreachable → silent data loss).

Race window: from the end of flush_core(sub)'s bottom-up (its parent blocktable update, hammer2_flush.c:1116-1134) to maintenance's combined_find/bcmp (chain.c:4179-4210) — the remainder of B's RB_SCAN plus B's own CRC/flush work. An unprivileged local user running append-write loops against files whose inode chains live under a collapsible indirect block races the syncer flush.

Experiments (guest, stock kernel #0)

1. run_df2645.sh 12 min mode 1: 6000-file dir, 70% pre-deleted, 4 append writers (~130M appends total), churn create/delete children, sync loop at 20 Hz. Result: no panic, clean umount (full_run.log). Post-mortem: the pre-stress sparsification syncs had already collapsed the sparse indirects, so maintenance had no eligible work during the race phase.
2. Same with continuous key-space-wide churn (full_run2.log): writers + create/delete churn filled the 512MB volume mid-run; dmesg filled with hammer2_flush: ... error=00000020 (HAMMER2_ERROR_ENOSPC from the chain_modify(parent) failure path, hammer2_flush.c:1057-1064) and the teardown sync wedged in h2coll (ENOSPC retry storm — DF-2633 family, not this finding). No LOST CHILD*, no debug repchange, no excessive loops markers: the maintenance skip was never taken with a live child before ENOSPC took over.

Why it stays likely (not demoted)

Every link in the chain is individually verifiable in source: the skip guard exists and is reachable with a non-deleted, rbtree-resident child (bcmp mismatch via COW modify); repchange unconditionally asserts emptiness; the call order guarantees the assert runs after any skip. What was not achieved is winning the scheduler interleaving inside the window on this guest within the time budget (and the volume-fill side effect of the churn harness sabotaged attempt 2).

Suggested fix

--- a/sys/vfs/hammer2/hammer2_chain.c
+++ b/sys/vfs/hammer2/hammer2_chain.c
@@ repchange caller (indirect_maintenance, chain.c:4226-4228)
-   hammer2_chain_unex ... spin release ...
-   hammer2_chain_repchange(parent, chain);
+   if (chain->core.live_count == 0 && RB_EMPTY(&chain->core.rbtree)) {
+       hammer2_chain_repchange(parent, chain);
+   } else {
+       /* children were skipped due to concurrent modification;
+        * leave the chain linked (not deleted from the parent's
+        * rbtree/blocktable) and let the next flush retry the
+        * collapse.  Requires undoing the chain delete at
+        * chain.c:4160 (defer it to this point). */
+       ...
+   }

The robust structure is to make the collapse loop's skip path count skips and, if any occurred, abort the collapse before hammer2_chain_delete(parent, chain) performs any destructive step (move the delete after a successful full sweep, or pre-flight-verify like the DF-2632 fix does for key-range overlaps).