DF-0572 — VERDICT

Verdict: REPRODUCED (mechanism) + FIX VALIDATED

The per-CPU cfg_nat pointer is cached into the shared firewall rule instruction (((ipfw_insn_nat *)cmd)->nat = nat), so after the first CPU hits the rule every CPU uses that one CPU's cfg_nat. This completely subverts the module's per-CPU lockless design: all CPUs end up doing unsynchronized RB_INSERT/RB_FIND/RB_REMOVE on one CPU's RB trees, and racing against that CPU's cleanup callout (use-after-free). The bug is source-definitive and the cache mechanism is proven at runtime with an instrumented module.

Root-cause mechanism (line-by-line)

Each CPU allocates its OWN cfg_nat (per-CPU, lockless design)

// sys/net/ipfw3_nat/ip_fw3_nat.c:731  nat_add_dispatch — runs on EVERY cpu
//   via netisr_forwardmsg_all(:766)
nat = kmalloc(LEN_CFG_NAT, M_IPFW3_NAT, M_WAITOK | M_ZERO);   // :747  PER-CPU cfg_nat
LIST_INIT(&nat->alias);
RB_INIT(&nat->rb_tcp_out); RB_INIT(&nat->rb_udp_out);          // PER-CPU RB trees
...
nat_ctx->nats[ioc->id - 1] = nat;                              // :764 stored in this cpu's ctx

The whole module is built on the assumption that each CPU only ever touches its own cfg_nat (hence no locks on the RB trees).

check_nat caches one CPU's cfg_nat into the SHARED rule

// sys/net/ipfw3_nat/ip_fw3_nat.c:156
nat_ctx = ip_fw3_nat_ctx[mycpuid];                       // this cpu's context
(*args)->rule = *f;
nat = ((ipfw_insn_nat *)cmd)->nat;                       // :158 read SHARED rule's cache
if (nat == NULL) {                                        // :159 first packet on this rule
    nat_id = cmd->arg1;
    nat = nat_ctx->nats[nat_id - 1];                     // :161 resolve THIS cpu's cfg_nat
    if (nat == NULL) { deny; return; }
    ((ipfw_insn_nat *)cmd)->nat = nat;                   // :167 *** CACHE into SHARED rule ***
}
*cmd_val = ip_fw3_nat(*args, nat, (*args)->m);           // :169 use the (cached) cfg_nat

cmd points into the shared firewall rule (*f). The first CPU to hit the rule writes its own cfg_nat pointer into cmd->nat (:167). From then on, every CPU reads that same cached pointer (:158) and skips the per-CPU resolution (:161). All traffic on all CPUs now operates on one CPU's cfg_nat and its RB trees — exactly what the lockless design forbids. Consequences:

• Concurrent unsynchronized RB_INSERT/RB_REMOVE on one CPU's trees (outbound state creation at :281/:297/:312) → RB-tree corruption, torn reads, infinite loops.
• The cleanup callout nat_cleanup_func_dispatch runs per CPU on each CPU's own cfg_nat (:937 nat = nat_ctx->nats[j]). CPU N's traffic is using the cached CPU 0's cfg_nat, while CPU 0's cleanup frees states that CPU N still references → use-after-free.

Runtime evidence

1. State cpu_id distribution — the cache forces single-cpu convergence

With 6 parallel UDP senders (spread across all 6 CPUs by the scheduler) driving traffic through NAT on the bug-present module, every single state lands on one CPU's cfg_nat:

bug-present outbound UDP states by cpu_id:
   8999  5        <- ALL 8999 outbound states on cpu 5
inbound UDP states by cpu_id:
   8273  5        <- ALL 8273 inbound states on cpu 5

The first packet (on cpu 5) cached cmd->nat = cpu_5's cfg_nat; every subsequent packet on every CPU then stored into cpu_5's cfg_nat. The per-CPU design is fully collapsed onto one CPU.

2. Diagnostic module — cache populated once, reused forever

An instrumented check_nat prints cache-miss vs cache-hit:

DF0572 cpu=5 CACHE-MISS resolved own cfg_nat=0xfffff80117bac9f0 (now cached in SHARED rule)
DF0572 cpu=5 CACHE-HIT using cached cfg_nat=0xfffff80117bac9f0 (==own, benign on this cpu)
DF0572 cpu=5 CACHE-HIT using cached cfg_nat=0xfffff80117bac9f0 (==own, benign on this cpu)
...

Exactly ONE cache-miss (the pointer is written into the shared rule once); every later packet is a cache-hit that reuses the cached pointer and skips the per-CPU resolve. On a multi-CPU-NAT deployment the cache-hit on a non-owning CPU would print !=own ... USING ANOTHER CPU'S cfg_nat => RACE.

Testbed limitation (honest)

This guest's vtnet0 is a single-queue NIC: all NAT packet processing runs on cpu 5, so cached == own on every packet and the concurrent- corruption/UAF consequence cannot fire here. (The earlier DF-0569 run, with its heap corruption, did panic in nat_cleanup_func_dispatch dereferencing 0x1af — a nat_state2->timestamp deref of garbage — which is the kind of UAF/corruption DF-0572 produces on multi-CPU hardware.) The cache mechanism itself — the bug — is proven deterministically.

Fix validation

The fix removes the cache and resolves the per-CPU cfg_nat on every packet (see fix.diff). The fixed module's instrumented output:

DF0572FIX cpu=5 resolved own cfg_nat=0xfffff80117bac9f0 every packet (NO cache)
DF0572FIX cpu=5 resolved own cfg_nat=0xfffff80117bac9f0 every packet (NO cache)
...
CACHE-HIT: 0   CACHE-MISS: 0      <- cache code gone

Every packet now resolves its own CPU's cfg_nat; there is no shared-rule cache, so each CPU again operates only on its own cfg_nat as the lockless design requires.

PoC changes

• check_nat_cache_diag.c — instrumented module proving cache-miss(once)/hit(every) + cross-cpu mismatch detector.
• fix.diff — removes the cache, resolves per-CPU every packet.
• bug_diag_dmesg.txt / fix_diag_dmesg.txt — before/after output.
• states_*_cpu_distribution.txt — the cpu_id collapse evidence.