DF-3399

Unreplied cross-CPU netmsg in NAT state-add path leaks the kmalloc'd message when dispatch CPU diverges from hash CPU

Summary

The state-add path kmallocs a netmsg_nat_state_add and netisr_sendmsg()s it, but nat_state_add_dispatch never calls netisr_replymsg - only netisr_afree_rport replies kfree; a reply to td_msgport never frees. One ~80B message leaked per new NAT flow whenever nextcpu != mycpuid. On the guest the branch provably never fires (udp_output re-dispatches to netisr_hashcpu before ip_output; same-hash for TCP/forwarded): 36k NAT'd flows left lwkt_message in-use flat at 2 - not_reproduced; reachable in principle when a NIC RSS hash diverges from ipfw3's software recompute. Fix: netmsg_init with netisr_afree_rport + netisr_replymsg.

Discussion (0)

No comments yet.

Severity: Low
Status: new
Bucket: Base system — network stack
Confidence: likely
Pass: 2 · GLM 5.3 second pass
CVSS 3.1: CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:N/I:N/A:L
CWE: CWE-401
File: sys/net/ipfw3_nat/ip_fw3_nat.c
Lines: 385-725
Area: net
Discovered: 2026-09-12
Updated: 2026-09-12

Team triage

untriaged

priority: — · owner: —

PoC verification

not_reproduced

Status: not_reproduced
Reproduced: no
Impact: none
Confidence: likely
Tested: 2026-09-12 06:02:47
Attempts: 3
Runtime: 420s
Code hash: —
Guest: DragonFly dfbsd 6.5-DEVELOPMENT DragonFly 6.5-DEVELOPMENT #0: Thu Jul 2 06:02:54 UTC 2026 root@dfbsd:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64
Build cmd: cc -O2 -o /root/udp_spray /root/udp_spray.c
Run cmd: vmstat -m | grep lwkt_message; su -m maxx -c /tmp/udp_spray\ 30000; vmstat -m | grep lwkt_message

jump to evidence pack ↓

Fix verification

not_testable

Fix status: not_testable
Baseline reproduced: —
Patched reproduced: —
Confidence: likely
Tested: 2026-09-12 06:02:47
Runtime: —
Kernel: per-fix-DF-3399
fix.diff: findings/poc/DF-3399/fix.diff
Runner: df-bsd-reviewer

jump to fix verification ↓

PoC verification

Evidence pack

findings/poc/DF-3399 · 3 files

File	Type	Size
udp_spray.c	—	1.3 KB	view raw
leak.sh	—	932 B	view raw
run.log	—	591 B	view raw

Fix verification

not_testable

per-fix-DF-3399

Confirmed kernel references

Detail

Evidence (decisive lines)

["findings/poc/DF-3399/run.log: vmstat -m lwkt_message Count stays 2 across 36k NAT'd flows (baseline + isolation spray)", 'sys/net/ipfw3_nat/ip_fw3_nat.c:384-407 netmsg kmalloc+sendmsg; :701-725 handler without netisr_replymsg/forwardmsg', 'sys/netinet/udp_usrreq.c:1136-1160 redispatch-to-hash-cpu before ip_output (why the guest never crosses CPUs)']

PoC changes

wrote udp_spray.c (fresh source port per datagram to force a new 5-tuple/state per send); leak.sh harness using vmstat -m per malloc-type

Verified recommended fix

netmsg_init with &netisr_afree_rport and terminate nat_state_add_dispatch with netisr_replymsg(&add_msg->base, 0)

Verdict

nat_state_add_dispatch (ip_fw3_nat.c:701-725) never replies or forwards the kmalloc'd netmsg_nat_state_add, which per the lwkt/netisr API (only netisr_afree_rport replies kfree the message, netisr.c:138-142) means the ~80B message can never be freed -- a per-new-flow kernel heap leak whenever the branch runs. On the 6-vCPU verification guest the branch could not be exercised: udp_output re-dispatches to netisr_hashcpu(udp_addrhash()) before ip_output (udp_usrreq.c:1136-1160) and forwarded packets are dispatched by the same ip_hashfn, so nextcpu == mycpuid deterministically; 36,000 NAT'd UDP flows left the lwkt_message malloc-type in-use count flat at 2 (run.log), and an isolation spray with the NAT rule deleted showed the per-flow Requests growth is the plain UDP path, not ipfw3. The leak is reachable in principle when a NIC RSS hash (M_HASH set by the driver) diverges from ipfw3's software recompute (ip_fw3_nat.c:379-380 clears M_HASH and recomputes), leaking one netmsg per new NAT flow -- gradual, root-config-gated memory exhaustion. Filed Low/likely: API violation certain, trigger narrow and not present on this guest.