# DF-3462 — ipfw3 `set toggle` only flips CPU0's per-CPU sets bitmap

**File:** `sys/net/ipfw3/ip_fw3_set.c`
**Verdict:** reproduced (policy-divergence logic bug, fix validated)
**Impact:** per-CPU firewall policy divergence — `ipfw3 set toggle N`
(= `IP_FW_SET_TOGGLE`, used by `ipfw3 set disable/enable` semantics)
takes effect only for flows hashed to CPU0; every other CPU keeps the
old enable/disable state forever.

## What to look at

- `run.log` — baseline (stock `ipfw3.ko`), the decisive four-phase run:
  - ENABLED-1: all 23 source aliases counted (delta 2 = OUT+IN hooks).
  - DISABLED-1 (`ipfw3 set show` ⇒ `disable: 3`): sources
    127.0.0.{4,8,13,19,20,24} → **delta 0** (rule skipped, CPU0-hashed),
    the other 17 sources → **delta 2** (rule STILL enforced — bug).
  - ENABLED-2 / DISABLED-2: the differential is reversible and
    deterministic per flow.
- `run.patched.log` — same script after installing the `fix.diff`
  module: DISABLED phases now yield **delta 0 for every source**
  (uniform skip across all 6 CPUs); ENABLED phases unchanged. The
  divergence is gone.
- `srcscan.sh` + `udp1shot.c` — the measurement instrument: one
  bound-source UDP datagram per 127.0.0.X alias to local 127.0.0.1;
  the flow CPU is `netisr_hashcpu(toeplitz(dst,src))`
  (`sys/netinet/ip_demux.c:71-80`, addresses only — ports ignored).
- `build.log` — module-only rebuild of `ipfw3.ko` with `fix.diff`
  (`cc 8.3 [DragonFly]`).
- `panic.teardown.txt` — an unrelated stock `kldunload ipfw3` crash in
  `rn_flush` observed during an earlier teardown (independent lead,
  not on the DF-3462 path).

## Reproduce (root on the guest)

```sh
sysctl net.filters_default_to_accept=1      # avoid deny-all on module load
kldload ipfw3 && kldload ipfw3_basic && kldload ipfw3_layer4
for x in 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24; do
    ifconfig lo0 inet 127.0.0.$x netmask 255.255.255.255 alias
done
cc -O -o /root/udp1shot udp1shot.c
ipfw3 add 104 set 3 count udp dst-port 44990
./run.sh          # baseline: DISABLED phases show delta 0 only for
                  # {4,8,13,19,20,24}; delta 2 for all others  <-- BUG
```

Fixed module:

```sh
cd /usr/src && patch -p1 < fix.diff
cd sys/net/ipfw3 && make && make install
<reboot or load the patched module from a clean boot>
./run.sh          # DISABLED phases now show delta 0 for ALL sources
```

Success criterion (baseline): with `disable: 3` reported by
`ipfw3 set show`, a strict non-empty subset of flows (CPU0-hashed) skips
set-3 rules while the majority still enforces them. After `fix.diff`:
uniform behavior. See `VERDICT.md` for the full mechanism and source
citations.