# DF-3462 VERDICT — reproduced; fix validated (module rebuild + rerun)

## Finding

`ip_fw3_ctl_set_toggle()` (`sys/net/ipfw3/ip_fw3_set.c:191-204`) mutates
only `fw3_ctx[mycpuid]->sets`. Every ipfw3 sockopt handler runs under
`ASSERT_NETISR0` (`sys/netinet/raw_ip.c:308`), so `mycpuid == 0` always:
the toggle flips **CPU0's bitmap only**. Per-packet rule matching,
however, runs on whichever CPU the flow hashes to and consults that
CPU's private context:

- `ip_fw3_chk()`: `ctx = fw3_ctx[mycpuid]` (`sys/net/ipfw3/ip_fw3.c:350`)
- rule skip: `if (ctx->sets & (1 << f->set)) continue;`
  (`sys/net/ipfw3/ip_fw3.c:487-489`) — applies to **any** action
  (deny / allow / count / skipto / fwd …).

The four sibling handlers in the same file broadcast their state change
to every CPU via `netisr_domsg`/`netisr_forwardmsg_all`
(`ip_fw3_set.c:114` move_set, `:148` move_rule, `:186` swap, `:250`
flush), and ipfw2's equivalent (`ipfw_set_disable_dispatch`,
`sys/net/ipfw/ip_fw2.c:5849-5875`) broadcasts as well — only the
toggle was missed. The consequence: after any odd number of
`ipfw3 set toggle N`, rules in set N are skipped for CPU0-hashed flows
and remain enforced for all others (deterministic per address pair —
`INP_MPORT_HASH_UDP` hashes addresses only, `sys/netinet/ip_demux.c:71-80`).
An operator who disables a set (emergency removal of bad deny rules,
incident response) sees `ipfw3 set show` / `ipfw3 -S list` confirm
"disabled" (both read CPU0's bitmap, `ip_fw3_set.c:211`,
`ip_fw3.c:1016`) while the set's rules keep firing on ~5/6 of the
traffic. There is no CPU where the reverse holds, because CPUs 1..N-1
never receive the bit at all.

## How it was reproduced (guest, 6 CPUs, DragonFly 6.5-DEVELOPMENT #0)

Observable: rule `104 set 3 count udp dst-port 44990`. `IP_FW_GET`
aggregates the per-CPU sibling counters (`ip_fw3.c:1020-1025`), so a
datagram is counted iff its checking CPU has set 3 enabled. Flow CPU
control: datagrams from bound source aliases 127.0.0.X to local
127.0.0.1; CPU = `netisr_hashcpu(toeplitz(127.0.0.1, 127.0.0.X))`.

Baseline (`run.log`, stock module):

| state (`ipfw3 set show`) | 127.0.0.{4,8,13,19,20,24} | other 17 sources |
|---|---|---|
| enabled (`disable:` empty) | delta 2 (counted) | delta 2 (counted) |
| `disable: 3`              | **delta 0 (skipped)** | **delta 2 (STILL enforced)** |
| toggle back               | delta 2                | delta 2 |

Deterministic, reversible, reproduced across four phases and three
independent experiment rounds (destination-side scan, source-side
scan, repeated runs). The skip branch used by `count` is the same
branch that skips `deny`/`allow` — the enforcement divergence applies
to every action in the set.

Dead ends honestly recorded: (1) an attempt to demonstrate the deny
direction through packet *delivery* was confounded by DFly's per-CPU
UDP wildcard-socket placement ("dropped due to no socket",
`netstat -s -p udp`) — unrelated to ipfw3, excluded from the claim;
(2) `nc -u -l` exits immediately when stdin closes — replaced by
`udprx.c`; (3) pinning the *sending process* does not steer the flow
CPU because `udp_send` redispatches by 4-tuple-address hash
(`sys/netinet/udp_usrreq.c:1137-1147`).

## Fix validation

`fix.diff` (authored after the baseline reproduction) broadcasts the
toggle exactly like the file's own `set_swap`/`flush` handlers:

```c
void
set_toggle_dispatch(netmsg_t smsg)
{
	struct netmsg_sets *msg = (struct netmsg_sets *)smsg;
	struct ipfw3_context *ctx = fw3_ctx[mycpuid];

	ctx->sets = ctx->sets ^ (1 << msg->set_to);
	netisr_forwardmsg_all(&smsg->base, mycpuid + 1);
}
```

Module-only rebuild + install in the guest (`build.log`), loaded from
a clean `vm.sh reset with-src` boot, identical test (`run.patched.log`):

| state | 127.0.0.{4,8,13,19,20,24} | other 17 sources |
|---|---|---|
| enabled | delta 2 | delta 2 |
| `disable: 3` | **delta 0** | **delta 0** |

Uniform behavior — divergence eliminated; enable direction unchanged.
`ip_fw3_ctl_set_get` needs no change once all bitmaps agree (it reads
CPU0's, which equals every CPU's after the broadcast).

## Unrelated incident

`kldunload ipfw3` (stock fini path) panicked once in `rn_flush` during
an early teardown — see `panic.teardown.txt`. Not on the DF-3462 code
path (the toggle handler is never invoked during module ops); kept as
an independent lead for the maintainers.

## Bottom line

Certain. Root-only configuration primitive, but a silent, deterministic
per-CPU firewall-policy divergence on a multi-CPU system: set
enable/disable (toggle) affects only CPU0-hashed flows. Medium severity
(firewall policy bypass/incorrect enforcement; requires the firewall
operator to use set toggles). Fix is three lines of broadcast + one
prototype, validated by module rebuild and differential re-run.

## Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest `make -j6 nativekernel KERNCONF=X86_64_GENERIC` completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.