DF-3462 — ipfw3 set toggle only flips CPU0's per-CPU sets bitmap

File: sys/net/ipfw3/ip_fw3_set.c Verdict: reproduced (policy-divergence logic bug, fix validated) Impact: per-CPU firewall policy divergence — ipfw3 set toggle N (= IP_FW_SET_TOGGLE, used by ipfw3 set disable/enable semantics) takes effect only for flows hashed to CPU0; every other CPU keeps the old enable/disable state forever.

What to look at

• run.log — baseline (stock ipfw3.ko), the decisive four-phase run:
• ENABLED-1: all 23 source aliases counted (delta 2 = OUT+IN hooks).
• DISABLED-1 (ipfw3 set show ⇒ disable: 3): sources 127.0.0.{4,8,13,19,20,24} → delta 0 (rule skipped, CPU0-hashed), the other 17 sources → delta 2 (rule STILL enforced — bug).
• ENABLED-2 / DISABLED-2: the differential is reversible and deterministic per flow.
• run.patched.log — same script after installing the fix.diff module: DISABLED phases now yield delta 0 for every source (uniform skip across all 6 CPUs); ENABLED phases unchanged. The divergence is gone.
• srcscan.sh + udp1shot.c — the measurement instrument: one bound-source UDP datagram per 127.0.0.X alias to local 127.0.0.1; the flow CPU is netisr_hashcpu(toeplitz(dst,src)) (sys/netinet/ip_demux.c:71-80, addresses only — ports ignored).
• build.log — module-only rebuild of ipfw3.ko with fix.diff (cc 8.3 [DragonFly]).
• panic.teardown.txt — an unrelated stock kldunload ipfw3 crash in rn_flush observed during an earlier teardown (independent lead, not on the DF-3462 path).

Reproduce (root on the guest)

sysctl net.filters_default_to_accept=1      # avoid deny-all on module load
kldload ipfw3 && kldload ipfw3_basic && kldload ipfw3_layer4
for x in 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24; do
    ifconfig lo0 inet 127.0.0.$x netmask 255.255.255.255 alias
done
cc -O -o /root/udp1shot udp1shot.c
ipfw3 add 104 set 3 count udp dst-port 44990
./run.sh          # baseline: DISABLED phases show delta 0 only for
                  # {4,8,13,19,20,24}; delta 2 for all others  <-- BUG

Fixed module:

cd /usr/src && patch -p1 < fix.diff
cd sys/net/ipfw3 && make && make install
<reboot or load the patched module from a clean boot>
./run.sh          # DISABLED phases now show delta 0 for ALL sources

Success criterion (baseline): with disable: 3 reported by ipfw3 set show, a strict non-empty subset of flows (CPU0-hashed) skips set-3 rules while the majority still enforces them. After fix.diff: uniform behavior. See VERDICT.md for the full mechanism and source citations.

DF-3462 VERDICT — reproduced; fix validated (module rebuild + rerun)

Finding

ip_fw3_ctl_set_toggle() (sys/net/ipfw3/ip_fw3_set.c:191-204) mutates only fw3_ctx[mycpuid]->sets. Every ipfw3 sockopt handler runs under ASSERT_NETISR0 (sys/netinet/raw_ip.c:308), so mycpuid == 0 always: the toggle flips CPU0's bitmap only. Per-packet rule matching, however, runs on whichever CPU the flow hashes to and consults that CPU's private context:

• ip_fw3_chk(): ctx = fw3_ctx[mycpuid] (sys/net/ipfw3/ip_fw3.c:350)
• rule skip: if (ctx->sets & (1 << f->set)) continue; (sys/net/ipfw3/ip_fw3.c:487-489) — applies to any action (deny / allow / count / skipto / fwd …).

The four sibling handlers in the same file broadcast their state change to every CPU via netisr_domsg/netisr_forwardmsg_all (ip_fw3_set.c:114 move_set, :148 move_rule, :186 swap, :250 flush), and ipfw2's equivalent (ipfw_set_disable_dispatch, sys/net/ipfw/ip_fw2.c:5849-5875) broadcasts as well — only the toggle was missed. The consequence: after any odd number of ipfw3 set toggle N, rules in set N are skipped for CPU0-hashed flows and remain enforced for all others (deterministic per address pair — INP_MPORT_HASH_UDP hashes addresses only, sys/netinet/ip_demux.c:71-80). An operator who disables a set (emergency removal of bad deny rules, incident response) sees ipfw3 set show / ipfw3 -S list confirm "disabled" (both read CPU0's bitmap, ip_fw3_set.c:211, ip_fw3.c:1016) while the set's rules keep firing on ~5/6 of the traffic. There is no CPU where the reverse holds, because CPUs 1..N-1 never receive the bit at all.

How it was reproduced (guest, 6 CPUs, DragonFly 6.5-DEVELOPMENT #0)

Observable: rule 104 set 3 count udp dst-port 44990. IP_FW_GET aggregates the per-CPU sibling counters (ip_fw3.c:1020-1025), so a datagram is counted iff its checking CPU has set 3 enabled. Flow CPU control: datagrams from bound source aliases 127.0.0.X to local 127.0.0.1; CPU = netisr_hashcpu(toeplitz(127.0.0.1, 127.0.0.X)).

Baseline (run.log, stock module):

Deterministic, reversible, reproduced across four phases and three independent experiment rounds (destination-side scan, source-side scan, repeated runs). The skip branch used by count is the same branch that skips deny/allow — the enforcement divergence applies to every action in the set.

Dead ends honestly recorded: (1) an attempt to demonstrate the deny direction through packet delivery was confounded by DFly's per-CPU UDP wildcard-socket placement ("dropped due to no socket", netstat -s -p udp) — unrelated to ipfw3, excluded from the claim; (2) nc -u -l exits immediately when stdin closes — replaced by udprx.c; (3) pinning the sending process does not steer the flow CPU because udp_send redispatches by 4-tuple-address hash (sys/netinet/udp_usrreq.c:1137-1147).

Fix validation

fix.diff (authored after the baseline reproduction) broadcasts the toggle exactly like the file's own set_swap/flush handlers:

void
set_toggle_dispatch(netmsg_t smsg)
{
    struct netmsg_sets *msg = (struct netmsg_sets *)smsg;
    struct ipfw3_context *ctx = fw3_ctx[mycpuid];

    ctx->sets = ctx->sets ^ (1 << msg->set_to);
    netisr_forwardmsg_all(&smsg->base, mycpuid + 1);
}

Module-only rebuild + install in the guest (build.log), loaded from a clean vm.sh reset with-src boot, identical test (run.patched.log):

Uniform behavior — divergence eliminated; enable direction unchanged. ip_fw3_ctl_set_get needs no change once all bitmaps agree (it reads CPU0's, which equals every CPU's after the broadcast).

Unrelated incident

kldunload ipfw3 (stock fini path) panicked once in rn_flush during an early teardown — see panic.teardown.txt. Not on the DF-3462 code path (the toggle handler is never invoked during module ops); kept as an independent lead for the maintainers.

Bottom line

Certain. Root-only configuration primitive, but a silent, deterministic per-CPU firewall-policy divergence on a multi-CPU system: set enable/disable (toggle) affects only CPU0-hashed flows. Medium severity (firewall policy bypass/incorrect enforcement; requires the firewall operator to use set toggles). Fix is three lines of broadcast + one prototype, validated by module rebuild and differential re-run.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.