DF-5839 / verdict.json
{ "finding_id": "DF-5839", "status": "reproduced", "reproduced": true, "impact": "panic", "confidence": "certain", "verdict": "Reproduced deterministically on the stock INVARIANTS guest (X86_64_GENERIC, 6 cpus): a foreign-cpu lwkt_hold() on a thread whose corpse has just been published to gd_freetd survives (per-cpu critical sections cannot exclude cross-cpu atomics), lwkt_alloc_thread() reuses the corpse without ever checking td_refs, lwkt_init_thread()s bzero wipes the live refcount (console: reborn child running on cpu5 refs=0), and the stale lwkt_rele() decrements the REBORN threads td_refs to -1 -> panic assertion \"td->td_refs > 0\" failed in lwkt_rele at lwkt_thread.c:505 (reproduced twice; run.2.log is an independent rerun on the same binary). On stock (non-INVARIANTS) kernels the same chain is silent: td_refs = -1 makes the reborn threads lwkt_exit() drain loop spin in tsleep(\"tdreap\", 1 tick) forever (permanent thread-exit livelock) and the rele is a write into recycled-then-reused kernel memory; the user-thread twin (lwp_wait/kern_exit.c:857 -> lwp_dispose -> lwkt_free_thread) frees a thread whose ref is still held. Reachability is the exported-API class shared with DF-2728/DF-2729 (KLD/driver calling lwkt_hold from a foreign cpu without migrating to the owner; all in-tree callers - kern_proc.c:1804-1818 migrates, uipc_msg.c:232 self-holds - follow the owner-cpu discipline). fix.diff (refuse gd_freetd corpses with td_refs != 0, fall back to a fresh objcache allocation) was built into a kernel on the guest, booted (uname #1 Tue Oct 13 23:10:23), and the identical trigger then completed cleanly: lwkt_alloc_thread: freetd corpse ... still has 1 refs, dropping, fresh td allocated, stale rele benign, no panic across all three trigger modes.", "exploit_chain": "KLD on cpu0 observes cpu1s gd_freetd publication of an exiting thread (lwkt_thread.c:1742) -> lwkt_hold(corpse) from the foreign cpu (refs 0->1, invisible to the owners crit section) -> any lwkt_alloc_thread on cpu1 (kthread_create et al) grabs the corpse checking only TDF_RUNNING|TDF_PREEMPT_LOCK|TDF_RUNQ (:366-367) -> lwkt_init_thread bzero wipes td_refs (:445) -> stale lwkt_rele atomically decrements the REBORN thread (:506) -> INVARIANTS: panic in lwkt_rele (:505); stock: td_refs=-1 -> reborn threads lwkt_exit livelocks forever in tsleep(\"tdreap\") and the decrement itself was a write-after-reuse on the recycled struct.", "evidence": [ "panic.txt / console-full.log: baseline panic 'assertion \"td->td_refs > 0\" failed in lwkt_rele at /usr/src/sys/kern/lwkt_thread.c:505', cpuid=0, backtrace rebirth_and_rele() <- watcher_func()", "run.log: full chain markers - 'foreign hold on cached corpse ... from cpu0 (refs=0 freetd=...)' -> 'spawner on cpu1 consuming gd_freetd=...' -> 'child td=... (== victim corpse: YES)' -> reborn child 'running on cpu5 refs=0'", "run.2.log: independent second baseline panic on the same binary (identical backtrace addresses)", "run.patched.log: fixed kernel - 'lwkt_alloc_thread: freetd corpse ... still has 1 refs, dropping', child != victim, no panic through modes 0/1/2", "env.txt: baseline+patched uname, patched kernel.stripped sha256, guard-string presence checks", "fix.diff: git-apply clean against sys/" ], "kernel_refs": [ "sys/kern/lwkt_thread.c:497-507", "sys/kern/lwkt_thread.c:1707-1742", "sys/kern/lwkt_thread.c:365-380", "sys/kern/lwkt_thread.c:445", "sys/kern/lwkt_thread.c:505-506", "sys/kern/lwkt_thread.c:1735", "sys/kern/kern_exit.c:857-860", "sys/kern/kern_proc.c:1804-1818" ], "poc_changes": "No seed PoC existed; trigger authored from scratch. Two fixes during bring-up: (1) DFly tsleep() returns EWOULDBLOCK (nonzero) on timeout, so the sysctl completion wait must not break on the tsleep return value; (2) the loader ignored kernel=\"kernel.TEST/kernel\" in loader.conf, so the fixed kernel had to be installed as /boot/kernel/kernel directly (the silent fallback doubled as a second baseline reproduction). The deterministic mode-1 design (watch gd_freetd == victim plus TDF_RUNNING clear before the foreign hold, then force consumption on cpu1) removes all timing sensitivity: the corpse sits in gd_freetd until consumed.", "attempts": 4, "guest_uname": "DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64", "runtime_sec": 3180, "guest_dirty": 0, "build_cmd": "sh build.sh (guest; module: cd /tmp/df5839 && make obj && make; fix kernel: cd /usr/src && patch -p1 < fix.diff && make -j6 nativekernel KERNCONF=X86_64_GENERIC)", "run_cmd": "kldload /usr/obj/tmp/df5839/df5839.ko && sysctl -n debug.df5839.run=0 && sysctl -n debug.df5839.run=1 && sysctl -n debug.df5839.run=2", "code_hash": "3f0ec86539ae2ee25522c540cf0aa76accbb30e8cc3b9159945c61da173ae03b", "notes": "INVARIANTS kernels panic (Low, KLD-class like DF-2728/DF-2729); stock kernels execute the corrupted state: silent td_refs=-1 -> permanent lwkt_exit livelock of the reborn thread + refcount write-after-reuse. Guest reset with-src (clean) after verification and fix validation.", "recommended_fix": "In lwkt_alloc_thread()s gd_freetd branch, refuse corpses with td_refs != 0 (kprintf diagnostic + fresh objcache_get) so lwkt_init_thread()s bzero can never wipe a live refcount; leaking the dropped corpses cached kstack is safe because the holder still owns the pointer.", "fix_status": "fixed", "fix_baseline_reproduced": true, "fix_patched_reproduced": false, "fix_verdict": "fix.diff applied inside the guest /usr/src, kernel rebuilt with nativekernel (BUILD_RC=0, kernel.stripped sha256 2abb1011cc2f284d1044f954aff2f3346f9e567ae4de78295336c1495ba8fc71), installed as /boot/kernel/kernel, booted (uname #1: Tue Oct 13 23:10:23 UTC 2026). Exact same PoC re-run: baseline panicked at lwkt_thread.c:505; patched kernel executes the identical trigger with the guard firing (lwkt_alloc_thread: freetd corpse 0xfffff800905e1400 still has 1 refs, dropping), allocates a fresh thread, and completes modes 0/1/2 with no panic and the guest up.", "fix_evidence": [ "run.patched.log: guard message + child != victim + no panic (modes 0/1/2)", "env.txt: patched-kernel boot stamp, kernel.stripped sha256, guard-string check", "VERDICT.md: full baseline-vs-patched narrative" ] } |