DF-2851 PoC — netmsg_so_notify abort race (accept path)

What it shows

An unprivileged user racing signal-aborted blocking accept(2) against concurrent TCP connection completions panics the stock INVARIANTS kernel inside netmsg_so_notify_abort() (sys/kern/uipc_msg.c) because soaccept_predicate() reassigns msg->base.nm_so while the notify message is still abortable, so the abort path locks/lists the wrong socket. On non-INVARIANTS kernels the same interleaving is a wrong-list TAILQ_REMOVE + double lwkt_replymsg (kernel heap corruption).

Build (inside guest, unprivileged ok)

cc -O2 -pthread -Wall -o /tmp/accept_abort_race /tmp/accept_abort_race.c

Run (inside guest, unprivileged)

/tmp/accept_abort_race 240

Expected

Kernel panic within seconds-to-minutes:

LWKT_RELTOKEN ASSERTION ...
lwkt_reltoken: release wants pool and got pool
panic: lwkt_reltoken: illegal release
lwkt_reltoken() at lwkt_reltoken+0xda
lwkt_reltoken() at lwkt_reltoken+0xda
netmsg_so_notify_abort() at netmsg_so_notify_abort+0xe9
netmsg_service_loop() at netmsg_service_loop+0xe9

(observe with vm.sh log). Observed: run #1 panicked in <20 s, run #2 (fresh reset) in ~2 min. With fix.diff applied and kernel rebuilt: no panic over 2x240 s.

Files

accept_abort_race.c   PoC source (listeners/acceptors/killers/connectors)
build.sh / run.sh     exact commands
build.log             compiler output
run.log               decisive run transcript (run #2)
panic.txt             run #1 panic (serial console)
panic_run2.txt        run #2 panic (serial console, fresh reset)
boot_full*.txt        full serial console captures
env.txt               guest environment
fix.diff              git-apply-able fix (validated in-guest)
VERDICT.md            full narrative
manifest.json         machine catalog
verdict.json          machine verdict

DF-2851 — netmsg_so_notify abort race: nm_so mutated by accept predicate → unserialized abort on wrong socket (wrong-list TAILQ_REMOVE / illegal token release / double reply)

Verdict in one paragraph

REPRODUCED (panic), unprivileged, 2/2 stress runs on the stock INVARIANTS kernel (#0 X86_64_GENERIC, 6 vCPU). An unprivileged local user racing a signal-aborted blocking accept(2) against concurrent connection completion drives netmsg_so_notify_abort() (sys/kern/uipc_msg.c:734) into operating on a different socket than the one it locked, because soaccept_predicate() reassigns msg->base.nm_so (uipc_syscalls.c:261) while the notify message is still queued and abortable. On the INVARIANTS guest this panics deterministically-enough (lwkt_reltoken: illegal release inside netmsg_so_notify_abort); on a production (non-INVARIANTS) kernel the same interleaving executes TAILQ_REMOVE() against the wrong socket's ssb_mlist and a second lwkt_replymsg() on an already-replied message — kernel heap list corruption with stale tqe_prev/tqh_last pointers plus a double reply. Fix validated: with the one-line root fix (dedicated nm_result output field, nm_so never mutated) the same stress runs clean for 2×240 s where the unfixed kernel panicked in ≤2 min.

Full narrative

The machinery

DragonFly's kern_accept() blocks on a stack-allocated netmsg_so_notify message (uipc_syscalls.c:293,345-350) dispatched via lwkt_domsg(head->so_port, ..., PCATCH) to the listener's netisr thread. netmsg_so_notify() (uipc_msg.c:652) runs the predicate under the head's pool token; if unsatisfied it queues the message on head->so_rcv.ssb_mlist and sets SSB_MEVENT.

Two independent parties can later complete (reply) the queued message:

1. netmsg_so_notify itself (same netisr thread that owns head->so_port) — safe vs. abort by port FIFO ordering.
2. sowakeup() (uipc_socket2.c:616-630) — the SSB_MEVENT scan — which runs on whatever CPU the connection completed on: soisconnected() → sorwakeup(head), explicitly noted as cross-CPU in uipc_socket2.c ("XXX head may be on a different protocol thread. sorwakeup()->sowakeup() is hacked atm."). For TCP the completing CPU is chosen by the 4-tuple hash — an attacker spraying connect(2) from different source ports steers it at will.

The defect

On the completion path, soaccept_predicate() reassigns the message's socket identity:

/* uipc_syscalls.c:255-262 */
soreference(so);
so->so_head = NULL;
...
msg->base.lmsg.ms_error = 0;
msg->base.nm_so = so;          /* <-- "Abuse nm_so field ... XXX JH" */
return (TRUE);

after which sowakeup() removes the message from head's mlist and calls lwkt_replymsg() (which sets MSGF_REPLY atomically).

Meanwhile the blocked acceptor can be signalled (PCATCH): lwkt_waitmsg → lwkt_abortmsg → netmsg_so_notify_doabort (uipc_msg.c:702) → abort message to head->so_port → netmsg_so_notify_abort (uipc_msg.c:734), which derives both its interlock token and its list from the mutable field:

/* uipc_msg.c:744-751 */
lwkt_getpooltoken(nmsg->base.nm_so);                 /* (a) token of X */
if ((nmsg->base.lmsg.ms_flags & (MSGF_DONE | MSGF_REPLY)) == 0) {
        ssb = (nmsg->nm_etype & NM_REVENT) ?
                &nmsg->base.nm_so->so_rcv :          /* (b) list of Y */
                &nmsg->base.nm_so->so_snd;
        TAILQ_REMOVE(&ssb->ssb_mlist, nmsg, nm_list);/* wrong list */
        lwkt_relpooltoken(nmsg->base.nm_so);
        lwkt_replymsg(&nmsg->base.lmsg, EINTR);      /* double reply */
}

lwkt_getpooltoken() is an address-hashed token pool (lwkt_token.c:816-823): head and the just-accepted so hash to different pool slots, so once nm_so has been reassigned the abort takes a token that does not serialize with the completing CPU's sowakeup (which holds head's slot across its predicate→remove→reply sequence). The DONE|REPLY recheck at uipc_msg.c:745 therefore races the completion's lwkt_replymsg, and every use of nm_so inside the critical section can observe a different socket than the one locked at (a).

Observed manifestations

• INVARIANTS guest (this PoC): nm_so changed between (a) and the release → the abort releases a pool token it never acquired: panic: lwkt_reltoken: illegal release, backtrace lwkt_reltoken ← lwkt_reltoken ← netmsg_so_notify_abort ← netmsg_service_loop (panic.txt, panic_run2.txt). Note that in the TRUE branch the TAILQ_REMOVE at uipc_msg.c:749 executes before the panicking release — the wrong-list unlink already happened.
• Non-INVARIANTS kernels: no token assertion fires. The abort (i) unlinks nmsg from accepted_so->so_rcv.ssb_mlist, a list it was never on — with standard TAILQ_REMOVE (no TRASHIT poisoning outside QUEUEDEBUG, sys/sys/queue.h:139) the stale tqe_prev/tqe_next from head's list are re-unlinked, corrupting tqh_last/links of one or both lists (subsequent netmsg_so_notify inserts and sowakeup scans then write through dangling pointers), and (ii) replies the message with EINTR racing the completion's own lwkt_replymsg (double reply — the KKASSERT at lwkt_msgport.c:595 exists precisely because double replies are fatal to the wait machinery).

Reproduction

• PoC: accept_abort_race.c (8 blocking acceptors on a loopback TCP listener, 2 killer threads delivering SIGUSR1 with sa_flags=0 — no SA_RESTART, so every delivery aborts the in-flight notify message — and 6 connector threads completing connections). All unprivileged (maxx, uid 1001).
• Run #1 (pre-reset guest): panic within 20 seconds.
• Run #2 (fresh vm.sh reset with-src boot): panic within ~2 minutes (see run.log, panic_run2.txt). 2/2 runs panicked.

Exploitability

• Who: any local unprivileged user (sockets + signals only).
• What: on non-INVARIANTS kernels, a cross-list TAILQ_REMOVE with stale pointers (write *(tqe_prev) = tqe_next into another socket's notify list, plus tqh_last corruption) and a double message reply. The corrupted ssb_mlist is exercised by subsequent accepts/wakeups (TAILQ_INSERT_TAIL writes through *tqh_last), giving a repeatable, partially-controlled write primitive into kernel heap objects — a genuine memory-corruption escalation surface, though racing it into a controlled uid=0 chain was not pursued within this run's budget (the guest panics long before the list corruption can be groomed on this INVARIANTS kernel). Impact ceiling: kernel memory corruption → privilege escalation; floor: reliable local DoS (panic).
• The window is soaccept_predicate's nm_so = so … completion's atomic_set(MSGF_REPLY) — tens of ns wide, but both event streams are user-paced and the stress hit it within minutes on 6 vCPUs.

Fix validation (see fix section of manifest)

• Baseline: unfixed kernel panics ≤2 min into the stress (2/2).
• Patched: fix.diff (nm_result output field; nm_so immutable) applied in the guest, make nativekernel KERNCONF=X86_64_GENERIC, reboot into the patched kernel, same PoC run 2×240 s: no panic, guest up, clean summary lines. The defect class is closed by making the abort path's lock/list identity immutable; the double-reply and wrong-list-remove interleavings cannot occur because nm_so no longer changes after queueing.

kernel_refs

• sys/kern/uipc_msg.c:702-715 (netmsg_so_notify_doabort reads nm_so)
• sys/kern/uipc_msg.c:744-751 (token + ssb + TAILQ_REMOVE + reply from mutable nm_so)
• sys/kern/uipc_syscalls.c:244-262 (soaccept_predicate reassigns base.nm_so)
• sys/kern/uipc_socket2.c:616-630 (sowakeup foreign-cpu MEVENT scan replies)
• sys/kern/uipc_socket2.c:239-283 (soisconnected → sorwakeup(head), "may be on a different protocol thread")
• sys/kern/lwkt_token.c:816-823 (pool tokens are address-hashed → no cross-socket serialization)
• sys/kern/lwkt_msgport.c:595 (double-reply KKASSERT), :184-203 (domsg sync contract)