lwkt_alloc_thread() gd_freetd rebirth wipes live td_refs: foreign-cpu lwkt_hold() in the lwkt_exit() publication window turns a stale lwkt_rele() into refcount write-after-reuse and permanent exit livelock

DF-5839 — lwkt_alloc_thread() gd_freetd rebirth wipes live td_refs: foreign-cpu lwkt_hold() in the lwkt_exit() publication window converts a stale lwkt_rele() into a write-after-reuse / permanent exit livelock ======================================================================

WHAT (bug) lwkt_exit() protects its final td_refs check and the gd_freetd publication (sys/kern/lwkt_thread.c:1707-1742) only with a per-cpu critical section. lwkt_hold()/lwkt_rele() (:497-507) are cross-cpu atomics that a critical section on the exiting cpu cannot exclude. A foreign-cpu lwkt_hold() landing after the check leaves a published gd_freetd corpse with td_refs == 1. lwkt_alloc_thread() (:365-368) grabs that corpse checking only TDF_RUNNING|TDF_PREEMPT_LOCK|TDF_RUNQ — never td_refs — and lwkt_init_thread() (:445) bzero()s the struct, wiping the refcount. The stale lwkt_rele() then decrements the REBORN thread's td_refs to -1: * INVARIANTS: panic "td->td_refs > 0" in lwkt_rele (:505), or "td->td_refs == 0" in lwkt_exit (:1735) when the hold lands inside the check->publish window. * stock: silent -1; when the reborn thread exits, lwkt_exit()'s drain loop spins in tsleep("tdreap", 1 tick) forever (permanent thread-exit livelock), and the rele wrote recycled-then-reused memory. Same reachability class as DF-2728/DF-2729 (users of the exported lwkt_hold/lwkt_rele API from KLDs/drivers; in-tree callers all follow the owner-cpu discipline — kern_proc.c:1804-1818 migrates first, uipc_msg.c:232 self-holds — so stock systems are not exposed by in-tree code).

HOW TO BUILD (guest, root) sh build.sh # KLD: cc -Werror ... -> /usr/obj/tmp/df5839/df5839.ko

HOW TO RUN (guest, root) kldload /usr/obj/tmp/df5839/df5839.ko sysctl -n debug.df5839.run=0 # control: owner-cpu hold/rele (clean) sysctl -n debug.df5839.run=1 # deterministic foreign hold on the # published gd_freetd corpse -> rebirth # -> stale rele sysctl -n debug.df5839.run=2 # racy hold inside the exit window

EXPECTED (baseline, stock INVARIANTS kernel — reproduced) mode 1: df5839: foreign hold on cached corpse from cpu0 (refs=0 freetd=) df5839: spawner on cpu1 consuming gd_freetd= df5839: child td= (== victim corpse: YES) df5839: child (reborn corpse) running on cpu5 refs=0 <-- bzero wiped panic: assertion "td->td_refs > 0" failed in lwkt_rele at /usr/src/sys/kern/lwkt_thread.c:505 cpuid = 0 ... Debugger("panic") (panic.txt, console-full.log)

EXPECTED (with fix.diff applied, kernel rebuilt & booted — verified) mode 1: lwkt_alloc_thread: freetd corpse still has 1 refs, dropping df5839: child td= (== victim corpse: no) victim td_refs now=0 -> NO panic, guest stays up (run.patched.log) mode 2: hold lands in the window, exit completes, no panic.

FIX VALIDATION fix.diff refuses gd_freetd corpses with td_refs != 0 (falls back to a fresh objcache allocation, leaking the corpse's cached kstack safely). Applied inside the guest's /usr/src, kernel rebuilt (make nativekernel KERNCONF=X86_64_GENERIC, BUILD_RC=0, kernel.stripped sha256 2abb1011cc2f284d1044f954aff2f3346f9e567ae4de78295336c1495ba8fc71), installed as /boot/kernel/kernel, rebooted (uname -> #1 Tue Oct 13 23:10:23), exact same PoC re-run: guard fires, no panic.

DF-5839 — VERDICT: reproduced (panic on INVARIANTS; stock consequence chain proven to the last contended step), fix validated on a rebuilt kernel.

BASELINE (stock INVARIANTS X86_64_GENERIC, 6 cpus, uname in env.txt)

Mode 1 (deterministic) executed the full chain on the live guest:

1. victim kthread homed on cpu1 exits via kthread_exit() -> lwkt_exit(); its corpse is published to cpu1's gd_freetd (lwkt_thread.c:1741-1742) with TDF_RUNNING cleared by lwkt_switch_return() during the final switch-out inside cpu_thread_exit() (vm_machdep.c:340-347).
2. watcher on cpu0 (FOREIGN cpu; per-cpu crit on cpu1 cannot exclude it) observes gd_freetd == victim and issues lwkt_hold(victim) (lwkt_thread.c:499 atomic_add_int) -> corpse published with td_refs=1. [run log: "foreign hold on cached corpse 0xfffff800905fee80 from cpu0 (refs=0 freetd=0xfffff800905fee80)"]
3. spawner kthread running on cpu1 calls kthread_create() -> lwkt_alloc_thread() (lwkt_thread.c:365-368) which grabs gd_freetd checking only TDF_RUNNING|TDF_PREEMPT_LOCK|TDF_RUNQ — td_refs is never consulted — and lwkt_init_thread() (lwkt_thread.c:445) bzero()s the struct, wiping td_refs 1 -> 0. [run log: "child td=0xfffff800905fee80 (== victim corpse: YES)"]
4. the reborn thread starts running: "child 0xfffff800905fee80 (reborn corpse) running on cpu5 refs=0" <-- live refcount gone.
5. the stale foreign lwkt_rele() (lwkt_thread.c:506 atomic_add_int -1) decrements the REBORN thread: 0 -> -1, caught by the INVARIANTS assert: panic: assertion "td->td_refs > 0" failed in lwkt_rele at /usr/src/sys/kern/lwkt_thread.c:505, cpuid=0, backtrace rebirth_and_rele() <- watcher_func(). (panic.txt; the serial lines are partially interleaved with the spawner's kprintf on cpu1 — both cpus were printing at once; the assertion text and :505 line are intact.)

Reproduced twice (the second run, run.2.log, was accidental: the first fix-validation boot silently fell back to the original kernel because the loader ignored kernel="kernel.TEST/kernel"; the identical panic and identical backtrace addresses on that run independently confirm the baseline). The very first "control" mode-0 runs (owner-cpu hold/rele, the kern_proc.c walker discipline) never panicked — three runs, system stable — proving the owner-cpu serialization is exactly what the protocol relies on and the foreign-cpu hold is what breaks it.

STOCK-KERNEL CONSEQUENCE (not runnable on this INVARIANTS guest — the asserts intercept — but mechanically proven up to that point by the trace above): the reborn thread carries td_refs = -1; when it later executes lwkt_exit(), the drain loop if (td->td_refs) (:1709) spins in tsleep("tdreap", 1 tick) forever — a permanent thread-exit livelock holding the kernel thread slot and its kstack — and step 5's atomic decrement was a write into recycled-then-reused kernel memory (the objcache object was live again as a different thread). The user-thread twin (lwp_wait() polling td_refs at kern_exit.c:857 -> lwp_dispose -> lwkt_free_thread) has the same window and ends in freeing a thread whose ref is still held.

REACHABILITY: exported-API class, same as DF-2728/DF-2729 — any KLD or driver holding a lwkt ref from a foreign cpu without migrating to the owner first. All in-tree callers follow the owner-cpu discipline (kern_proc.c:1804-1818 migrates; uipc_msg.c:232 self-holds), so stock systems are not exposed by in-tree code; INVARIANTS/debug kernels panic on the first misuse.

FIX VALIDATION (fix.diff — refuses gd_freetd corpses with td_refs != 0)

• guard added in lwkt_alloc_thread()'s gd_freetd branch; git-apply clean against sys/; applied inside the guest's /usr/src.
• kernel rebuilt: make -j6 nativekernel KERNCONF=X86_64_GENERIC, BUILD_RC=0; kernel.stripped installed as /boot/kernel/kernel (sha256 2abb1011...ba8fc71, uname "#1: Tue Oct 13 23:10:23 UTC 2026").
• exact same PoC re-run (module rebuilt against the patched tree): mode 1: "lwkt_alloc_thread: freetd corpse 0xfffff800905e1400 still has 1 refs, dropping" -> fresh td allocated -> stale rele hit the still-allocated leaked corpse (1->0, benign). NO panic; guest stayed up. mode 2 (window race): hold landed at refs=1 with freetd already published and TDF_RUNNING still set; exit completed; NO panic.
• baseline vs patched: baseline panics at lwkt_thread.c:505; patched executes the identical trigger with the guard refusing the rebirth. Fix status: fixed.

ARTIFACTS panic.txt serial console excerpt of the baseline panic console-full.log full serial console of the baseline session run.log decisive baseline reproduction (annotated) run.2.log second baseline panic (loader-fallback rerun) run.patched.log fixed-kernel run: guard fires, no panic, modes 0/1/2 build.log module build (full, clean, -Werror) env.txt uname / ncpu / kernel sha256 / patched-kernel checks df5839.c, Makefile, build.sh, run.sh, fix.diff

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.