DF-3456 — pf anchor walk-stack race → rule bypass (PoC evidence pack)

What this proves

pf_anchor_stack[64] (sys/net/pf/pf.c:148) is a single global array used as the anchor-walk state by pf_step_into_anchor()/pf_step_out_of_anchor(). The pf pfil hooks (pf_check_in/out, pf_ioctl.c:3199+) hold pf_token only shared, and DragonFly lwkt shared tokens allow concurrent holders on multiple CPUs (sys/kern/lwkt_token.c:726+). Two CPUs evaluating anchor rules concurrently therefore write and read the same frames, tearing each other's walk. A torn frame makes the victim walk resume the rule list at the other walk's position — skipping every rule in between, including block drop rules.

Observable result on the guest: UDP datagrams that pf is configured to block drop inside an anchor are delivered to userland, while an identical block in the main ruleset (control) drops 100%.

Files

• anchor_bypass.c — lo UDP flooders + blocked-port receivers (unprivileged)
• dns_flood.c — DNS query workers (unprivileged); their vtnet0-bound UDP query output walks are the concurrent walk population that makes the lo-output walks tear (see VERDICT.md "attribution")
• setup_pf.sh — loads the sandwich ruleset (run as root in the guest)
• run.sh — host-side driver (uses dfbsd-qemu/vm.sh)
• fix.diff — caller-local anchor walk stacks (verified fix)
• run.6.log — REPRODUCED (117 bypassed datagrams + counter forensics)
• run8_supplement.log — second reproduction (1 bypassed datagram)
• env.txt, build.log, kbuild.log — environment / build evidence

Ruleset shape (why it works)

pass all no state
anchor "spam" {
    <300 udp decoys>
    anchor "inner1" { pass udp port 9998 }   # walked as spam/spam/inner1
    block drop udp 127.0.0.1 -> 127.0.0.1 port 5555   # sandwiched block
    block drop udp 10.0.2.2 port 53
    anchor "inner2" { pass udp port 9999 }   # walked as spam/spam/inner2
    <300 tail decoys>
}
anchor "ham" { pass udp port 9997 }          # walked as ham
block drop udp 127.0.0.1 -> 127.0.0.1 port 5556    # control (main ruleset)

Walk A enters inner1 (frame[1] = {rs=spam, r=rule-inner1}) and walks its 1000 decoys. Concurrent walk B, slightly ahead, exits inner1, passes the block, and enters inner2 (frame[1] = {rs=spam, r=rule-inner2}). When A pops its frame it reads B's frame and resumes spam's list at TAILQ_NEXT(rule-inner2) — past both block rules. A's packet is then decided by pass all → delivered.

Two pfctl quirks had to be worked around (documented in VERDICT.md): 1. empty inline anchors are optimized away by pfctl (placeholder rule needed) 2. an inline anchor "inner1" inside anchor "spam" is walked by the kernel as the doubled path spam/spam/inner1; the walked decoys are loaded there with pfctl -a spam/spam/inner1 -f.

Build & run

On the host, from the repo root:

sh findings/poc/DF-3456/run.sh 90

The driver copies the pack into the guest, compiles as the unprivileged user maxx, sanity-checks the harness (probe with pf off), loads the ruleset as root, starts receivers as maxx, floods for 90 s, then prints both counters.

Success criterion

Receiver on 5555 (block inside anchor) reports > 0 datagrams, receiver on 5556 (identical block in main ruleset) stays 0, and per-rule evaluation counters show the sandwich block lagging the surrounding decoys (tear/skip signature).

DF-3456 — VERDICT

Status: REPRODUCED — pf anchor-walk frame tearing delivers packets that pf is configured to block drop, with a clean same-ruleset control.

Root cause

• pf_anchor_stack[64] is a single global array (sys/net/pf/pf.c:143-148) used as the per-walk anchor recursion state by pf_step_into_anchor() (pf.c:2501-2533) and pf_step_out_of_anchor() (pf.c:2535-2570). asd (the depth) is local to each caller, but the frames are shared by all CPUs.
• The pf data path takes pf_token only shared: pf_check_in/out, pf_check6_in/out (sys/net/pf/pf_ioctl.c:3199-3263) wrap pf_test() in lwkt_gettoken_shared(&pf_token). DragonFly shared tokens are genuine shared/exclusive reader-writer tokens (sys/kern/lwkt_token.c:726-785, _lwkt_trytokref() shared branch), so two or more CPUs can execute pf_test() → pf_test_rule()/pf_match_translation()/pf_test_fragment() concurrently, each pushing/popping the same global frames.
• A walk that pops a frame written by another CPU resumes the rule list at TAILQ_NEXT(other walk's rule) — skipping every rule between the two anchor rules. When a block drop sits in that span, the verdict falls back to the earlier pass match → the packet is delivered.

Note the config side (pfioctl, DIOCXCOMMIT etc.) takes pf_token exclusively, so rule/anchor freeing is properly serialized against walkers — the defect is strictly shared-vs-shared, i.e. walker vs walker.

How it was demonstrated (guest: 6 vCPU DragonFly 6.5-DEVELOPMENT #0)

Ruleset (v6, setup_pf.sh): pass all no state, anchor spam containing 300 decoys, nested anchor inner1 (1000 decoys in the walked path), two adjacent block drop rules (udp 127.0.0.1:5555 and udp 10.0.2.2:53), nested anchor inner2 (10 decoys), 300 tail decoys; plus anchor ham (500 decoys) and a control block drop udp port 5556 in the main ruleset.

Traffic (unprivileged user maxx): 6 lo UDP flooders (5555/5556 alternating) + 24 DNS query workers (UDP to the slirp gateway 10.0.2.2:53 — these queries' output walks go out via vtnet0 and form the concurrent walk population), 90 s.

Run 6 (first reproduction) — run.6.log

• Receiver on 5555: 117 datagrams delivered through block drop ... port 5555.
• Receiver on 5556 (control, blocked in main ruleset): 0 — the control proves pf was up, filtering loopback, and dropping correctly everywhere except the anchor-sandwich block.
• Counter forensics: first decoy in spam 1,649,386 evaluations vs block-5555 1,628,593 — ~20,800 walks skipped the block rule entirely; inner2 first decoy 1,628,558 (consistent with resume-after-inner2).

Run 8 (second reproduction) — run8_supplement.log

• 1 datagram delivered on 5555, control 0.
• Counter gap grew by ~8,700 skips during the run (9,314,540 vs 9,304,616).

Run 7

Receivers were blind (port bind collision with a previous run's receivers — harness error), but the counter gap still grew by ~700 during the flood, showing the corruption occurred in all three instrumented configurations.

Control (attribution)

The same ruleset and flood with lo flooders only (no DNS workers) showed no observable delivery and ~zero counter-gap growth (gap 549 of 5.1M walks ≈ 0.01%) over 90 s. The DNS workers' ~250 vtnet0-bound UDP walks/s are what turn the lo-output walk population into a tearing one (cross-context/cross-CPU concurrent walkers). The bypass rate in the reproducing configuration was ~1.3% of walks torn and 117 delivered packets in 90 s in run 6.

Impact assessment

• Silent, intermittent firewall rule bypass under concurrent load on any pf deployment whose ruleset uses anchors (the standard way pf.conf files are structured with includes/tables): packets matching block/drop rules that sit after a nested anchor get passed.
• The corruption is of kernel control state; the walk stays inside valid kernel objects (rules/rulesets), so a direct memory-safety fault was not observed and appears structurally unreachable from pure frame tearing (every frame field is a valid kernel pointer even when torn).
• Trigger: unprivileged traffic alone does it once pf is enabled with the anchor ruleset loaded (loading rules needs root once, standard setup).

Fix (validated)

fix.diff removes the global array and gives each of the three walker functions (pf_match_translation, pf_test_rule, pf_test_fragment) a caller-local struct pf_anchor_stackframe astack[PF_ANCHOR_STACKSIZE] passed through the step functions' signatures. (A per-CPU array would not be airtight: a netisr thread can preempt a user-thread walk on the same CPU.) Kernel-stack cost: 2 KiB per walker frame (DF x86_64 kernel stacks are 4 pages/16 KiB).

Fix validation (patched kernel built in-guest with make nativekernel, rebooted, exact same reproducer): see the fix section in verdict.json — zero deliveries on 5555 and zero counter-gap growth on the patched kernel.

Auxiliary observations (not the security finding)

1. pfctl inline nested-anchor paths are doubled by the kernel: an inline anchor "inner1" inside anchor "spam" { ... } is loaded with anchor_call "spam/inner1" which the kernel resolves relative to the containing anchor → the walked anchor is spam/spam/inner1, while pfctl -a spam/inner1 -f writes to spam/inner1. Rules written the natural pfctl way inside nested inline anchors are silently unreachable (0 evaluations). pfctl/kernel disagreement, usability/correctness bug.
2. During early experiments, an unrelated 20k-rule single-anchor ruleset flooded by a single sender wedged the whole guest (sshd unresponsive, no panic on serial console). Not chased further; the 3k-rule rulesets used for the actual PoC never wedged the guest.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.