# DF-0572 — VERDICT

## Verdict: REPRODUCED (mechanism) + FIX VALIDATED

The per-CPU `cfg_nat` pointer is **cached into the shared firewall rule
instruction** (`((ipfw_insn_nat *)cmd)->nat = nat`), so after the first CPU
hits the rule every CPU uses that one CPU's `cfg_nat`. This completely
subverts the module's per-CPU lockless design: all CPUs end up doing
unsynchronized `RB_INSERT`/`RB_FIND`/`RB_REMOVE` on one CPU's RB trees, and
racing against that CPU's cleanup callout (use-after-free). The bug is
**source-definitive** and the cache mechanism is **proven at runtime** with
an instrumented module.

---

## Root-cause mechanism (line-by-line)

### Each CPU allocates its OWN cfg_nat (per-CPU, lockless design)
```c
// sys/net/ipfw3_nat/ip_fw3_nat.c:731  nat_add_dispatch — runs on EVERY cpu
//   via netisr_forwardmsg_all(:766)
nat = kmalloc(LEN_CFG_NAT, M_IPFW3_NAT, M_WAITOK | M_ZERO);   // :747  PER-CPU cfg_nat
LIST_INIT(&nat->alias);
RB_INIT(&nat->rb_tcp_out); RB_INIT(&nat->rb_udp_out);          // PER-CPU RB trees
...
nat_ctx->nats[ioc->id - 1] = nat;                              // :764 stored in this cpu's ctx
```
The whole module is built on the assumption that **each CPU only ever
touches its own `cfg_nat`** (hence no locks on the RB trees).

### check_nat caches one CPU's cfg_nat into the SHARED rule
```c
// sys/net/ipfw3_nat/ip_fw3_nat.c:156
nat_ctx = ip_fw3_nat_ctx[mycpuid];                       // this cpu's context
(*args)->rule = *f;
nat = ((ipfw_insn_nat *)cmd)->nat;                       // :158 read SHARED rule's cache
if (nat == NULL) {                                        // :159 first packet on this rule
    nat_id = cmd->arg1;
    nat = nat_ctx->nats[nat_id - 1];                     // :161 resolve THIS cpu's cfg_nat
    if (nat == NULL) { deny; return; }
    ((ipfw_insn_nat *)cmd)->nat = nat;                   // :167 *** CACHE into SHARED rule ***
}
*cmd_val = ip_fw3_nat(*args, nat, (*args)->m);           // :169 use the (cached) cfg_nat
```

`cmd` points into the **shared** firewall rule (`*f`). The first CPU to hit
the rule writes its own `cfg_nat` pointer into `cmd->nat` (:167). From then
on, **every CPU** reads that same cached pointer (:158) and skips the
per-CPU resolution (:161). All traffic on all CPUs now operates on **one
CPU's** `cfg_nat` and its RB trees — exactly what the lockless design
forbids. Consequences:

* Concurrent unsynchronized `RB_INSERT`/`RB_REMOVE` on one CPU's trees
  (outbound state creation at :281/:297/:312) → RB-tree corruption, torn
  reads, infinite loops.
* The cleanup callout `nat_cleanup_func_dispatch` runs **per CPU** on each
  CPU's own `cfg_nat` (:937 `nat = nat_ctx->nats[j]`). CPU N's traffic is
  using the cached CPU 0's `cfg_nat`, while CPU 0's cleanup frees states
  that CPU N still references → **use-after-free**.

---

## Runtime evidence

### 1. State cpu_id distribution — the cache forces single-cpu convergence

With 6 parallel UDP senders (spread across all 6 CPUs by the scheduler)
driving traffic through NAT on the **bug-present** module, every single
state lands on one CPU's `cfg_nat`:

```
bug-present outbound UDP states by cpu_id:
   8999  5        <- ALL 8999 outbound states on cpu 5
inbound UDP states by cpu_id:
   8273  5        <- ALL 8273 inbound states on cpu 5
```

The first packet (on cpu 5) cached `cmd->nat = cpu_5's cfg_nat`; every
subsequent packet on every CPU then stored into cpu_5's `cfg_nat`. The
per-CPU design is fully collapsed onto one CPU.

### 2. Diagnostic module — cache populated once, reused forever

An instrumented `check_nat` prints cache-miss vs cache-hit:
```
DF0572 cpu=5 CACHE-MISS resolved own cfg_nat=0xfffff80117bac9f0 (now cached in SHARED rule)
DF0572 cpu=5 CACHE-HIT using cached cfg_nat=0xfffff80117bac9f0 (==own, benign on this cpu)
DF0572 cpu=5 CACHE-HIT using cached cfg_nat=0xfffff80117bac9f0 (==own, benign on this cpu)
...
```
Exactly ONE cache-miss (the pointer is written into the shared rule once);
every later packet is a cache-hit that reuses the cached pointer and skips
the per-CPU resolve. On a multi-CPU-NAT deployment the cache-hit on a
non-owning CPU would print `!=own ... USING ANOTHER CPU'S cfg_nat => RACE`.

### Testbed limitation (honest)
This guest's `vtnet0` is a **single-queue** NIC: all NAT packet processing
runs on cpu 5, so `cached == own` on every packet and the concurrent-
corruption/UAF consequence cannot fire here. (The earlier DF-0569 run, with
its heap corruption, did panic in `nat_cleanup_func_dispatch` dereferencing
`0x1af` — a `nat_state2->timestamp` deref of garbage — which is the kind of
UAF/corruption DF-0572 produces on multi-CPU hardware.) The cache mechanism
itself — the bug — is proven deterministically.

---

## Fix validation

The fix removes the cache and resolves the per-CPU `cfg_nat` on every
packet (see `fix.diff`). The fixed module's instrumented output:
```
DF0572FIX cpu=5 resolved own cfg_nat=0xfffff80117bac9f0 every packet (NO cache)
DF0572FIX cpu=5 resolved own cfg_nat=0xfffff80117bac9f0 every packet (NO cache)
...
CACHE-HIT: 0   CACHE-MISS: 0      <- cache code gone
```
Every packet now resolves its own CPU's `cfg_nat`; there is no shared-rule
cache, so each CPU again operates only on its own `cfg_nat` as the lockless
design requires.

## PoC changes
* `check_nat_cache_diag.c` — instrumented module proving cache-miss(once)/hit(every) + cross-cpu mismatch detector.
* `fix.diff` — removes the cache, resolves per-CPU every packet.
* `bug_diag_dmesg.txt` / `fix_diag_dmesg.txt` — before/after output.
* `states_*_cpu_distribution.txt` — the cpu_id collapse evidence.