free_bw_list() runs unschedule_bw_meter() on bw_meter_timers[] without mroute_token (del_mfc / del_bw_upcall DELETE_ALL) β race vs bw_meter_process panic/UAF
Summary
del_mfc and del_bw_upcall(DELETE_ALL) detach the meter list under mroute_token, release the token, then call free_bw_list(), whose unschedule_bw_meter() walks and unlinks from the global bw_meter_timers[] hash with no lock. The 1 Hz bw_meter_process() callout (token held) disconnects whole buckets and reschedules entries; a colliding free-walk trips panic('unschedule_bw_meter: bw_meter entry not found') or operates on a meter the callout concurrently reschedules/kfrees (UAF write linking freed chunks into the hash). FreeBSD performs this free under the routing lock. Guest: race never landed in ~15 min of hammering (inconclusive - the deleter's token-taking detach serializes behind the callout's walk); mechanics documented line-by-line. Privileged trigger; DF-0306 family. Fix: keep the token across the free (FreeBSD order) - validated by build+boot and 45s no-regression run.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3303 Β· 10 files| File | Type | Description | Size | |
|---|---|---|---|---|
| bwrace.c | β | 5.6 KB | view raw | |
| ip_mroute_common.h | β | 2.2 KB | view raw | |
| build.sh | β | 95 B | view raw | |
| run.sh | β | 93 B | view raw | |
| build.log | β | 311 B | view raw | |
| run.log | β | 409 B | view raw | |
| fix_run.log | β | 151 B | view raw | |
| VERDICT.md | β | 2.9 KB | β raw | |
| fix.diff | β | 2.7 KB | view raw | |
| verdict.json | β | 3.2 KB | view raw |
DF-3303 VERDICT β INCONCLUSIVE on guest (race certain by inspection)
Bottom line
The locking violation is unambiguous in the code: del_mfc() and
del_bw_upcall(BW_UPCALL_DELETE_ALL) call free_bw_list() after
releasing mroute_token, and free_bw_list() β unschedule_bw_meter()
walks and unlinks from the global bw_meter_timers[] hash with no token
(ip_mroute.c:1115-1118, :2365-2368, :2653-2672) while the 1 Hz
bw_meter_process() callout (token held, :2684-2766) disconnects and
relinks whole buckets. A colliding free-walk either trips
panic("unschedule_bw_meter: bw_meter entry not found") (:2663) or
operates on memory the callout is concurrently rescheduling (UAF).
FreeBSD performs this exact free under the routing lock.
Six escalating attempts on the guest (up to 240 s each: 4 adder threads + 4 deleter threads at 10 ms + traffic suppression of the upcall storm; earlier variants with 120k-300k meter bursts) did not land the race:
- the deleter's detach (
mfc_find+ list detach) needsmroute_token, which the callout holds for its whole bucket walk β so deletes serialize behind the walk instead of overlapping it; - the free walk itself is fast (schedule pushes bucket head, and the mfc list is walked newest-first), leaving only a ~0.1-1% duty-cycle window per 1 Hz tick for the required interleaving.
Observed collateral during the attempts (recorded, not filed separately):
- bw_upcalls_send() (:2571-2578) does MGETHDR then m_copyback of
up to BW_UPCALLS_MAX * sizeof(struct bw_upcall) = 10 KB into a header
mbuf with no cluster: m_copyback prints
m_copyback: unexpected mbuf expansion required, code path needs to be
fixed and the upcalls are truncated β every 128 prepared upcalls when
thresholds are zero. Real bug (silent upcall loss + console spam), Low.
- meters left scheduled after the daemon dies churn the 1 Hz callout
forever (self-sustaining upcall storm, minutes-long console flood, ssh
starvation) β robustness note.
Verdict
inconclusive: reproduced=0 on the guest in the time available; the
finding stands on code inspection (confidence: likely; manifestation
timing-dependent). Trigger is privileged (MRT_DEL_MFC /
MRT_DEL_BW_UPCALL), so severity Medium at most, matching the DF-0306
family profile.
Fix (validated by build+boot, no crash regression in 45 s bwrace run)
Move the frees under the token (see fix.diff):
list = mfc->mfc_bw_meter;
mfc->mfc_bw_meter = NULL;
- lwkt_reltoken(&mroute_token);
free_bw_list(list);
+ lwkt_reltoken(&mroute_token);
(and the same reordering in del_mfc).
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
inconclusiveFix builds and boots; bwrace 45s no-regression run clean. Since the baseline race never landed on the guest, fix efficacy against the race itself is proven by construction (free path now under the same token as the timer walk), not by differential crash testing.
fix_build.log; fix_run.log
Confirmed kernel references
Detail
Evidence (decisive lines)
['run.log (bwrace phases, no panic)', 'fix_run.log (45s on patched kernel, no regression)', 'VERDICT.md (full attempt history and mechanics)']
PoC changes
Multiple redesigns: v1 dedupe bug (identical meters collapse to one - add_bw_upcall dedupes); v2 burst design starved the box with upcall-storm console spew; v3 traffic-suppressed thresholds + continuous add/del threads. Final: 4 adders + 4 delers@10ms + trafficker.
Verified recommended fix
Keep mroute_token held across free_bw_list() in del_mfc and del_bw_upcall (FreeBSD order); optionally assert token-hold inside unschedule_bw_meter.
Verdict
Code-certain lock-discipline race: del_mfc (:1115-1118) and del_bw_upcall DELETE_ALL (:2365-2368) run free_bw_list() after lwkt_reltoken(), and unschedule_bw_meter() (:2653-2672) mutates the global bw_meter_timers[] hash unlocked, racing the token-holding 1 Hz bw_meter_process() (:2715-2761) which disconnects and relinks whole buckets; collision => panic('unschedule_bw_meter: bw_meter entry not found') (:2663) or UAF reschedule of a freed meter. Six escalating guest attempts (up to 240s, 4 adder + 4 deler threads at 10ms, traffic-suppressed upcall storm, 120k-300k-meter burst variants) did not land the interleaving: the deleter's detach needs the token the callout holds across its walk, and free walks are fast (low duty cycle vs the 1 Hz tick). FreeBSD frees this list under the lock. Recorded collateral: bw_upcalls_send MGETHDR+m_copyback(10KB) fails ('m_copyback: unexpected mbuf expansion required') losing upcalls and flooding the console; scheduled meters outliving their daemon churn the callout forever.
No comments yet.