DF-2921 — INT64 overflow in nvtruncbuf/nvnode_pager_setsize → unkillable kernel livelock

Unprivileged 3-syscall trigger on HAMMER2 that spins a CPU inside the kernel forever (unkillable), blocks all other access to the file's vnode (rm hangs, shutdown hangs — guest only recoverable by killing QEMU).

Build (in guest)

./build.sh            # cc -O2 -o /tmp/op2921 op2921.c

Run (as unprivileged user, THROWAWAY VM ONLY)

./run.sh              # backgrounds the trigger, shows R0/D state after 15s
# then: kill -9 <pid>  — has no effect on a vulnerable kernel

Expected on a VULNERABLE kernel

- op2921 stuck in kernel (state R0), no output past (at most) the
  first two lines; kill -9 ineffective
- second run on the same file parks in D state
- `rm /home/maxx/df2921.bin` hangs
  (console: `cache_lock_shared: rm blocked ... "df2921.bin"`)
- `shutdown -p now` hangs

Expected on a FIXED kernel (fix.diff applied)

All four steps of op2921 print OK and the process exits immediately
(see run.fixed.log).

Files

op2921.c              trigger
livelock-evidence.txt observations + exact overflow math
fix.diff              fix (sys/kern/vfs_vm.c: clamp + unsigned math)
run.fixed.log         trigger on patched kernel
VERDICT.md            full narrative

DF-2921 — signed overflow → permanent unprivileged kernel livelock (unkillable)

Verdict (baseline, stock kernel): REPRODUCED

File: sys/kern/vfs_vm.c (nvtruncbuf() line 149, nvnode_pager_setsize() lines 463-466, unmap loop 486-495) Class: integer overflow (CWE-190) → infinite loop (CWE-835) → DoS Impact: permanent, unkillable kernel livelock on one CPU per trigger, plus permanent hang of every other thread touching the same vnode (including rm, shutdown); guest becomes unshut-downable. Confidence: certain — reproduced live on the guest as an unprivileged user; source-level math exact; kernel compiled with -fno-strict-overflow so the wraparound is the implemented behavior.

Trigger (unprivileged, HAMMER2, 3 syscalls — see op2921.c)

ftruncate(fd, 0x7ffffffffffff000LL);   /* extend to 2^63-4096 (sparse) */
pwrite(fd, "A", 1, 0x7fffffffffffefffLL); /* dirty the last 64K block   */
ftruncate(fd, 0x7fffffffffff8000LL);   /* truncate mid-64K-block        */

HAMMER2 imposes no va_size clamp (sys/vfs/hammer2/hammer2_vnops.c:512-525) and passes nblksize = HAMMER2_PBUFSIZE = 64K to nvtruncbuf() (hammer2_vnops.c:1260-1263, hammer2_calc_logical() always returns HAMMER2_PBUFSIZE, sys/vfs/hammer2/hammer2_subr.c:276).

Observed on the guest

1. Process enters kernel and spins in state R0 forever (first run pid 1025, observed > 5 minutes, still R0).
2. kill -9 ineffective (the loop never returns to userland; lwkt_yield() does not process signals).
3. A second process touching the same file blocks forever (D5) behind the spinner's vp->v_token (taken at vfs_vm.c:153) and vm_object lock.
4. rm of the file hangs: serial console shows [diagnostic] cache_lock_shared: rm blocked on 0xfffff8008f54dc00 "df2921.bin".
5. Three shutdown -p now attempts all hung; guest only recoverable by killing QEMU. A crash dump was taken via debug.panic=1 (vmcore.0).

Root cause math

• nvtruncbuf() vfs_vm.c:148-151: truncloffset = length + (blksize - boff) = 0x7fffffffffff8000 + (65536 - 32768) = 0x8000000000000000 → INT64_MIN. The RB_SCAN compare (vfs_vm.c:273) then matches every buffer on the vnode (including negative-loffset metadata buffers on UFS) and destroys them with B_INVAL|B_NOCACHE — silently discarding dirty buffers.
• nvnode_pager_setsize() vfs_vm.c:461-466: nobjsize = OFF_TO_IDX(truncboffset + blksize + PAGE_MASK) = OFF_TO_IDX(0x8000000000000fff) — the sum wraps negative and the arithmetic >> yields vm_pindex_t 0xFFF8000000000000.
• Unmap loop vfs_vm.c:486-495: pi starts at ~2^51 and the loop runs to 0xFFF8000000000000 — ~1.8×10^19 iterations of vm_page_lookup_busy_wait() (RB miss) + lwkt_yield() ≈ years, holding the vnode token and the object lock the whole time.

The extend path overflows the same expression (vfs_vm.c:464 via nvextendbuf() at line 385) making object->size garbage-huge, which is harmless until the later truncate hits the loop.

Fix validation

fix.diff (against sys/kern/vfs_vm.c): clamp truncloffset to OFF_MAX, compute nobjsize/pi in unsigned 64-bit. Kernel rebuilt; re-run of the identical trigger completes instantly — see run.fixed.log.