lwkt_tokref published before initialization β DDB ps token walk (db_dump_td_tokens) can dereference a stale/NULL tr_tok (snapshot race)
Summary
lwkt_gettoken/shared/try advance td_toks_stop BEFORE _lwkt_tokref_init fills tr_tok; db_dump_td_tokens (run by DDB 'ps' for every thread) walks [base,stop) of DDB-frozen CPUs and derefs tr_tok with no guard - a thread frozen inside the ~3-instruction window presents a stale/zero tr_tok -> tok->t_ref NULL deref -> nested DDB entry/wedge. Window ~ns and must coincide with the stop-IPI freeze (~1e-6/CPU/entry) - honestly speculative/untested; positive control live (ps walks tokens normally). init-before-publish reorder + NULL-guard walker fix proposed.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-5729 Β· 5 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 981 B | β raw | |
| VERDICT.md | β | 3.4 KB | β raw | |
| fix.diff | β | 1.3 KB | view raw | |
| manifest.json | β | 982 B | view raw | |
| verdict.json | β | 3.1 KB | view raw |
DF-5729 β lwkt_tokref published before init; DDB ps token walk can deref it
Read VERDICT.md for the full analysis. Summary: lwkt_gettoken (and
lwkt_gettoken_shared, lwkt_trytoken) advance td->td_toks_stop BEFORE
_lwkt_tokref_init() fills ref->tr_tok (sys/kern/lwkt_token.c:660-663).
DDB's ps β db_dump_td_tokens (sys/ddb/db_ps.c:193-197) walks
[td_toks_base, td_toks_stop) of frozen threads and dereferences
ref->tr_tok with no guard. A thread frozen inside the window yields a
stale (valid β misprint) or zero (β NULL-deref fault inside DDB β
trap_fatal/kdb_trap wedge family) tr_tok.
Status: speculative/untested β the window is nanoseconds wide and DDB's
stop-IPI must land inside it; not drivable on demand on this rig. Positive
control: ps in a DDB session on the same kernel walks tokens normally
(see ../DF-5728/postfix-run.log).
fix.diff: reorder all three publishers to init-then-publish + NULL-guard
in db_dump_td_tokens.
DF-5729 β DDB db_dump_td_tokens can dereference an unpublished lwkt_tokref (snapshot race)
Status: SPECULATIVE β source-proven ordering flaw, nanosecond window, not driven live
Root cause
sys/kern/lwkt_token.c:658-663 (lwkt_gettoken; identical pattern in
lwkt_gettoken_shared :729-733 and lwkt_trytoken :795-799):
ref = td->td_toks_stop;
KKASSERT(ref < &td->td_toks_end);
++td->td_toks_stop; /* publish FIRST */
cpu_ccfence();
_lwkt_tokref_init(ref, tok, td, TOK_EXCLUSIVE|TOK_EXCLREQ); /* init AFTER */
The thread's token-ref list is published (++td_toks_stop) before the
ref's tr_tok/tr_owner/tr_count are initialized. Any snapshot reader
that walks [td_toks_base, td_toks_stop) between the two stores observes a
slot containing stale bytes: either a leftover (valid) pointer from a
previously released token, or zeros for a never-used slot.
sys/ddb/db_ps.c:184-202 (db_dump_td_tokens, reached from the ps
command via db_ps.c:149/172 for every thread on the runq/allq) is exactly
such a snapshot reader, and it dereferences without a guard:
for (ref = &td->td_toks_base; ref < td->td_toks_stop; ++ref) {
tok = ref->tr_tok; /* db_ps.c:194 β no NULL check */
db_printf(" %p[tok=%p", ref, ref->tr_tok);
if (tok->t_ref && td == tok->t_ref->tr_owner) /* db_ps.c:197 β deref */
DDB freezes all other CPUs at entry (CPUx stopping CPUs ... stopped in
every session log). If a CPU is halted inside the 3-instruction window of
lwkt_gettoken β or, equivalently, the sampled thread's td_toks_stop
points at an uninitialized slot β the ps walk reads tr_tok == NULL
(fresh array slot) and dereferences tok->t_ref at a small offset:
NULL-deref page fault inside DDB, which routes through
trap_fatal β kdb_trap (sys/platform/pc64/x86_64/trap.c:420,1096) β
the known DDB-wedge family (DF-2241 class): nested DDB re-entry / wedge of
the session. A stale non-NULL tr_tok merely prints a wrong token
(cosmetic).
Why untested (honest)
The window is ++stop; fence; 3 stores β a few ns. DDB's stop-IPI must
land with a CPU exactly inside it. With tokens acquired continuously by
every subsystem, the per-entry probability is on the order of 1e-6 or less
per CPU; driving it on demand is not practical on this rig. The positive
control (healthy ps walk in DDB on the same kernel, DF-5728 postfix
session) behaves normally.
Fix
fix.diff: (a) reorder all three publishers to init-then-publish (the
memory-ordering-safe direction: the ref is fully initialized before any
snapshot reader can see it β external visibility timeline unchanged);
(b) belt-and-braces NULL-guard in db_dump_td_tokens for any other
transient state. The (a) hunk should get upstream review for the
cpu_ccfence() placement relative to _lwkt_tokref_init's stores.
Severity
Low / speculative. Same console-actor threat model as the other DDB findings; worst case is a DDB-context page fault (wedge/panic), best case a misprinted token.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testablefix.diff authored post-verification against the read-only sys/ tree (git apply --check clean); not rebuilt/run - the nanosecond race cannot be triggered on demand to demonstrate before/after.
findings/poc/DF-5729/fix.diff
Confirmed kernel references
Detail
Evidence (decisive lines)
VERDICT.md (full source trace with path:line); fix.diff (applies clean); ../DF-5728/postfix-run.log (healthy ps TOKENS walk - positive control)
PoC changes
No seed PoC; none written - the race window (~3 instructions) cannot be hit deterministically from the DDB console rig.
Verified recommended fix
Initialize the tokref before publishing td_toks_stop (all three gettoken variants) and skip NULL tr_tok refs in db_dump_td_tokens (fix.diff).
Verdict
Source-proven ordering flaw, not driven live: lwkt_gettoken/lwkt_gettoken_shared/lwkt_trytoken publish td_toks_stop BEFORE _lwkt_tokref_init() initializes the ref's tr_tok (sys/kern/lwkt_token.c:660-663, 731-733, 797-799). db_dump_td_tokens (sys/ddb/db_ps.c:193-197), reached from DDB 'ps' for every runnable thread (db_ps.c:149,172), walks [td_toks_base, td_toks_stop) of CPUs frozen by DDB's stop-IPI and dereferences ref->tr_tok unguarded. A thread frozen inside the ~3-instruction window presents a stale tr_tok (valid leftover pointer -> wrong token printed, cosmetic) or zeros in a never-used slot (-> tok->t_ref deref of NULL+small offset -> page fault inside DDB -> trap_fatal/kdb_trap nested re-entry/wedge, the DF-2241 consequence family). The window is nanoseconds wide; the stop-IPI must land inside it, so on-demand reproduction is impractical - filed as speculative. Positive control: 'ps' in a live DDB session on the same kernel walks tokens normally (../DF-5728/postfix-run.log TOKENS lines). Fix: init-then-publish reorder at all three sites plus a NULL-guard in the walker (fix.diff, applies clean against sys/).
No comments yet.