DF-5800 — vm_map_delete() freehint hole_start miscomputed in the no-predecessor branch

Pass-3 (fresh-perspective) re-audit of sys/vm/vm_map.c. Severity Info (certain code defect, no reachable security impact). Evidence pack includes the three guest-live stress rigs run for this file's pass 3 as a whole.

Build / run (stress rigs — pass-3 dynamic evidence)

# on the guest, as an unprivileged user:
cc -O -Wall -o stackstress stackstress.c -lpthread
cc -O -Wall -o fragstress fragstress.c -lpthread
cc -O -Wall -o forkcow     forkcow.c     -lpthread
./fragstress && ./forkcow && ./stackstress

Expected (observed on DragonFly 6.5-DEVELOPMENT guest, see run.log): - fragstress: 3 rounds x ~3.1M mixed mprotect/madvise/minherit/mlock/munlock ops on a 2048-entry fragmented map, "complete, no corruption". - forkcow: 60 rounds of fork/COW/wired/shared/NONE-inherit churn + partial munmap/remap, "complete, 0 unexpected bytes". - stackstress: 40 rounds of concurrent main-stack growth vs a mutator thread (mprotect PROT_NONE/RWX, madvise, minherit) + setrlimit (RLIMIT_STACK) churn; child SIGSEGVs are userland-by-design; guard-page harness ends "SIGSEGV at guard as expected"; kernel stays up.

The defect (static)

sys/vm/vm_map.c:3250-3263 (vm_map_delete), the branch taken when the range start is not mapped:

} else {
        if (first_entry) {                       /* predecessor exists */
                entry = vm_map_rb_tree_RB_NEXT(first_entry);
                if (entry == NULL)
                        hole_start = first_entry->ba.start;
                else
                        hole_start = first_entry->ba.end;   /* correct */
        } else {                                 /* no predecessor      */
                entry = RB_MIN(vm_map_rb_tree, &map->rb_root);
                if (entry == NULL)
                        hole_start = vm_map_min(map);       /* correct */
                else
                        hole_start = vm_map_max(map);       /* WRONG: should
                                                               be vm_map_min */
        }
}

With hole_start = vm_map_max, the trailing vm_map_freehint_hole(map, hole_start, entry->ba.start - hole_start) (vm_map.c:3390-3393) computes an underflowed (huge) length and the update condition scan->length <= length && scan->start > start (vm_map.c:707-711) can never hold (scan->start > vm_map_max is never true), so the freehint table is not lowered even though a hole extending to the bottom of the map was (re)created. A later same-shape vm_map_findspace (length+align exact match, vm_map.c:1512-1516) starts its scan from the stale, higher hint and can, in principle, return "no space" while lower address space is free.

Why it is not reachable with impact (pass-3 trace)

• sys_munmap requires the entire range to be mapped (no holes) via vm_map_check_protection(..., VM_PROT_NONE, ...) at sys/vm/vm_mmap.c:586-590 (hole rejection at sys/vm/vm_map.c:3457). A syscall-originated delete therefore always satisfies vm_map_lookup_entry(map, start, ...) and takes the first branch (vm_map.c:3246-3249), which computes hole_start = start correctly.
• The buggy branch is exercised only by internal whole-map removes whose start is below every entry: exec (sys/kern/kern_exec.c:965, vm_map_remove(map, 0, VM_MAX_USER_ADDRESS)), vmspace_terminate (sys/vm/vm_map.c:499-500), and vmspace_entry_delete (sys/vm/vm_vmspace.c:653-654). Those operate on maps that are being emptied or destroyed; an empty (or dying) map always finds space from a stale hint, so no user-visible ENOMEM-with-free-space was constructible.

Verdict

Not reproduced as an impact (by construction, see VERDICT.md). The code defect itself is certain; recommend the one-line correctness fix in fix.diff. No kernel rebuild performed for an Info/no-impact finding.

VERDICT — DF-5800 (pass-3 re-audit of sys/vm/vm_map.c)

Status: not_reproduced (impact: none). The code defect is certain; no reachable impact could be constructed, and the pass-3 dynamic rigs (guest-live) showed the map machinery stable under the adversarial interaction classes enumerated below.

The defect, precisely

vm_map_delete() (sys/vm/vm_map.c:3224-3398) computes hole_start for the trailing vm_map_freehint_hole() update. In the sub-branch where the delete range starts below every entry in the map (vm_map.c:3257-3263: first_entry == NULL, RB_MIN(...) != NULL), hole_start is set to vm_map_max(map) instead of vm_map_min(map). The subsequent call at vm_map.c:3390-3392 (vm_map_freehint_hole(map, hole_start, entry->ba.start - hole_start)) then underflows its length argument, and the per-slot update condition at vm_map.c:708 (scan->length <= length && scan->start > start) is unsatisfiable because scan->start > vm_map_max never holds. Net effect: the freehint optimization silently fails to record bottom-of-map holes for the affected (length, align) slots.

Why no impact is reachable

1. sys_munmap (sys/vm/vm_mmap.c:544-595) rejects any range with a hole (vm_map_check_protection at :586; hole check at sys/vm/vm_map.c:3457-3460). Hence every munmap-driven vm_map_delete enters with start mapped and takes the lookup-succeeds branch (vm_map.c:3246-3249, hole_start = start), which is correct.
2. The buggy branch requires start below all entries. The only callers producing that are internal full-map removes: exec (kern_exec.c:965), vmspace_terminate (vm_map.c:499-500), vmspace_entry_delete (vm_vmspace.c:653-654). All of them leave the map empty or destroy the vmspace; stale hints cannot cause a wrong allocation (hints only raise the search start, and vm_map_findspace re-validates every candidate against the RB tree; vm_map_insert re-validates bounds independently at vm_map.c:1275-1296). Worst theoretical outcome is a spurious ENOMEM-while-space-free, which we could not construct from any syscall path.

Pass-3 dynamic evidence (all as unprivileged user, guest up throughout)

Ran three rigs targeting the interaction classes the pass-3 brief named:

Kernel: DragonFly dfbsd 6.5-DEVELOPMENT #0 Thu Jul 2 06:02:54 UTC 2026 (X86_64_GENERIC). Guest stayed up; dmesg clean of panics.

Negative result notes (classes hunted and killed in pass 3)

• vm_map_stack *addrbos + max_ssize wrap (vm_map.c:4042,4058): unreachable — sys_mmap strips user MAP_STACK (vm_mmap.c:429-439); sole caller is exec with fixed USRSTACK-maxssiz and kernel-controlled sizes (kern_exec.c:902,991); exec paths leave the map empty.
• vm_map_simplify_entry ignores the aux union (vm_map.c:1728-1780): for stacks the surviving entry keeps its own avail_ssize; growth chains preserve the reservation floor; merges can only shrink the recorded reservation (conservative). master_pde variant is confined to VM_MAPTYPE_VPAGETABLE, which kern_mmap gates behind vm.vkernel_enable (vm_mmap.c:179-186; =0 on this guest, verified).
• ONEMAPPING + ref>1 page-strip on munmap (vm_map.c:3345-3366): every ref 1→2 transition on the embedded ba goes through vm_map_backing_replicated which clears OBJ_ONEMAPPING when ref_count > 1 (vm_map.c:3510-3519) — including the insert-coalesce path (:1353-1368 + :1414); vm_map_clean's temp refs are read-lock-serialized against delete.
• vm_map_protect(set_max=TRUE): no in-tree caller passes TRUE (all four call sites FALSE) — no max_protection raise from userland.
• vm_map_growstack insert-coalesce + clip_end re-split (:4246-4263): traced line-by-line; the re-split reproduces the expected fragment, so the "Bad stack grow start/end" panic is unreachable; stale avail_ssize on the original stack entry after middle-segment munmap is capped by the monotonic vm_ssize RLIMIT accounting (:4206,4217-4220) — conservative direction only (also in DF-0940's domain).
• IN_TRANSITION protocol: eflags equality in simplify (:1718,1737,1765) excludes IN_TRANSITION neighbors from merging; vm_map_delete waits (:3282-3289); vm_map_lookup partition path waits (:4625-4631); unclip callers use CLIP_CHECK_BACK/FWD (:2656,2693,2886,2918).
• mlock/munlock wired_count/USER_WIRED accounting symmetric incl. failure backout and kernel-wired (VPAGETABLE) entries (:2606-2740, vm_fault_unwire decrement-once semantics).
• Reserve accounting: per-syscall clip consumption ≤2 vs MAP_RESERVE_COUNT=4; vmspace_fork reserves nentries+4 and consumes ≤1/old-entry (:3722-3723); td_nest_count guard at :4456-4461.
• vm_map_submap NULL-entry clip on empty map (:2140-2143): only caller is kmem_suballoc (boot-time, kernel map) — not user reachable.
• pmap_copy is a no-op on x86_64 (pmap.c:5648-5651) — fork cannot leak wired PTEs into the child pmap.
• MADV_SETMAP arbitrary master_pde (:2377-2406): vpte walk stays within the mapping's own object pages (vm_fault_vpagetable frames are object pindexes, masked lookups), and the whole maptype is gated behind vm.vkernel_enable=0 on this guest.

Fix

One-liner in fix.diff (vm_map_min in the no-predecessor sub-branch). Not kernel-validated: Info severity, no reachable impact, and a full kernel rebuild for a hint-table no-op fix is disproportionate; the fix is mechanically obvious (mirrors the entry == NULL sibling branch).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.