netgraph7 item+message leak: NG_FREE_ITEM() only sets a flag, so ng_address_hook/path/ID() error paths permanently leak the queue item and its entire control message; ~4096 leaked items permanently exhaust the netgraph objcache and wedge the system (objcache_get M_WAITOK sleeps forever)
Summary
netgraph7's NG_FREE_ITEM(item) only sets a never-tested NGQF_FREE flag; the item is physically freed only when the reference managed by ng_snd_item()/ng_apply_item() is dropped. Freshly-packaged items that fail addressing before ng_snd_item() (refs==0) are never freed: ng_address_hook/path/ID error paths all leak the item plus everything attached (struct ng_mesg up to ~20KB, or an mbuf chain). Reachable driver: ngc_send - every failing control-socket sendto() to a bogus path. Items come from the bounded objcache (maxalloc=4096); after ~4096 leaks the cache is exhausted forever and objcache_get(M_WAITOK) blocks non-interruptibly. REPRODUCED on guest (netgraph7 modules built+loaded): 3k failing sends leak ~2795 pages matching 3000 msgs; 50k and 250k runs both ended with 'Warning: objcache(netgraph) exhausted on cpu0!' and permanent guest wedge; 250k successful control sends leaked nothing. Root-gated control sockets (SYSCAP_RESTRICTEDROOT) with an unpriv race-dependent variant via data sockets. Fix validated (ng_free_item at refs==0 on the three error sites): 100k failing sends flat-memory, good-path regression clean. Sibling code already works around the mark-only semantics manually - design defect confirmed.
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3474 Β· 12 files| File | Type | Description | Size | |
|---|---|---|---|---|
| README.md | β | 4.9 KB | β raw | |
| VERDICT.md | β | 4.6 KB | β raw | |
| trigger.c | β | 4.0 KB | view raw | |
| build.sh | β | 204 B | view raw | |
| run.sh | β | 241 B | view raw | |
| run.log | β | 320 B | view raw | |
| run.2.log | β | 318 B | view raw | |
| wedge_serial.log | β | 2.2 KB | view raw | |
| fix_run.log | β | 450 B | view raw | |
| fix.diff | β | 890 B | view raw | |
| verdict.json | β | 5.9 KB | view raw | |
| manifest.json | β | 1.2 KB | view raw |
DF-3474 β netgraph7 NG_FREE_ITEM() does not free: ng_address_*() error paths leak the item + full control message (permanent objcache exhaustion / kernel wedge)
Where
sys/netgraph7/netgraph/ng_base.c
NG_FREE_ITEM(item)(defined insys/netgraph7/netgraph.h:816-820) only sets theNGQF_FREEflag (and NULLs the variable in debug builds). Nothing in the entire tree ever testsNGQF_FREEβ grep confirms the flag is vestigial.- The only places an item's memory is actually reclaimed are
ng_unref_item()(netgraph2.h:77-89, refcount drops to 0) reached at the end ofng_apply_item()(ng_base.c:2109) β i.e. only for items that are "in flight" throughng_snd_item()/ng_apply_item()β plus four manualng_unref_item()/ng_free_item()calls that bluetooth node authors added themselves (ng_btsocket_l2cap.c:1731,ng_btsocket_l2cap_raw.c:360,ng_btsocket_hci_raw.c:618,ng_hci_evnt.c:317). -
Therefore any caller that packages a fresh item (refs == 0) and then hits an
NG_FREE_ITEM()in an address-failure path leaks the item and everything attached to it, permanently: -
ng_address_hook()error pathng_base.c:3062-3069(item + mbuf chain) ng_address_path()error pathng_base.c:3094-3098(item +struct ng_mesg+ payload)ng_address_ID()error pathng_base.c:3117-3122(item +struct ng_mesg+ payload)
The reachable driver for ng_address_path() is the netgraph7 control-socket
send path: ng_socket.c:309-316 (ngc_send() calls ng_address_path() and
on failure just goto release β no manual free, unlike its sibling
ng_connect_data() at ng_socket.c:776 which does call ng_free_item()).
Impact
Each failed sendto() on an AF_NETGRAPH control socket to a non-existent path
leaks sizeof(struct ng_item) plus the whole control message
(kmalloc(len+1, M_NETGRAPH_MSG), up to ~20KB, user-chosen). Two compounding
ceiling effects, both reproduced on the guest:
- The item cache is
objcache_create_mbacked(M_NETGRAPH, ..., maxalloc=4096)(ng_base.c:2772). After ~4096 leaked items the objcache is exhausted forever (items are never returned), and every subsequentobjcache_get(ng_oc, M_WAITOK)blocks in a non-interruptiblessleep("objcache_get")(kern/kern_objcache.c:547-552). The kernel printsWarning: objcache(netgraph) exhausted on cpu0!and every thread touching netgraph wedges permanently. - Message payloads leak unboundedly (kmem exhaustion) if sends continue.
Observed twice on the guest: a plain root userland loop of failing control sends wedged the whole system (ssh dead, serial console silent after the objcache warning) β permanent until reboot.
Privilege / reachability
- Deterministic trigger: root (netgraph7 control sockets require
caps_priv_check(SYSCAP_RESTRICTEDROOT)atng_socket.c:182). - Race-dependent variant for unprivileged users:
ngd_attach(data socket) has no privilege check;NG_SEND_DATA_FLAGSinngd_send(ng_socket.c:474) funnels intong_address_hook(), whose failure path leaks item+mbuf when the hook/peer dies concurrently. - NOTE: the stock DragonFly module set ships the older
sys/netgraphstack; this defect is in thesys/netgraph7stack (loaded when the netgraph7 modules are the live ones, e.g. alongside the netgraph7 bluetooth code). The PoC builds/loads the netgraph7 modules to test it.
Reproduce (guest, root)
# one-time: build the netgraph7 modules in-guest cd /usr/src/sys/netgraph7/netgraph && make obj && make cd /usr/src/sys/netgraph7/socket && make obj && make mkdir -p /root/ng7 cp /usr/obj/usr/src/sys/netgraph7/netgraph/netgraph.ko /root/ng7/ cp /usr/obj/usr/src/sys/netgraph7/socket/ng_socket.ko /root/ng7/ # (reboot first if the OLD netgraph stack was ever loaded β the # AF_NETGRAPH domain registration makes ng_socket.ko unloadable) kldload /root/ng7/netgraph.ko && kldload /root/ng7/ng_socket.ko cc -O2 -Wall -o /root/trigger /root/df3474/trigger.c sysctl -n vm.stats.vm.v_free_count # before /root/trigger good 250000 # control: NO growth sysctl -n vm.stats.vm.v_free_count # after-good (~unchanged) /root/trigger bad 50000 # leak: wedges guest within seconds
Expected:
- good phase: 250k successful sends, free-page count unchanged.
- bad phase: each sendto returns ENOENT; free pages drop by ~4KB per send;
after ~4096+ sends the console prints
Warning: objcache(netgraph) exhausted on cpu0! and the system wedges
permanently (ssh dead). Guest must be force-reset.
(NB: netgraph7 expects NG_VERSION 8; stock userland headers carry the old
stack's NG_VERSION 2 β the trigger sets version 8 explicitly.)
Fix
fix.diff (validated in-guest, see VERDICT.md): free fresh items via
ng_free_item() when item->refs == 0, keep the mark-only NG_FREE_ITEM()
for items that are in flight through ng_apply_item().
DF-3474 β VERDICT
status: reproduced / impact: dos / confidence: certain β fix validated (fixed)
What was run
Guest: DragonFly 6.5-DEVELOPMENT #0 (X86_64_GENERIC, INVARIANTS), stock
kernel. The stock /boot/kernel netgraph modules are built from the OLD
sys/netgraph tree (verified: strings /boot/kernel/ng_socket.ko contains
ng_send_msg, and netgraph.ko lacks netgraph_msg) β so the netgraph7
stack under audit was built from the identical in-guest source
(md5-verified against the workspace tree) and loaded:
cd /usr/src/sys/netgraph7/netgraph && make obj && make cd /usr/src/sys/netgraph7/socket && make obj && make kldload netgraph7/netgraph.ko ; kldload netgraph7/ng_socket.ko
(the old stack must not have been loaded since boot β the AF_NETGRAPH domain registration makes ng_socket.ko permanently unloadable, so module-swap requires a reboot; a fresh boot loads nothing netgraph-related by default).
One quirk worth recording: netgraph7's kernel expects NG_VERSION 8 while
the stock userland <netgraph/ng_message.h> (old tree) says 2 β the
trigger sets version 8 explicitly or every sendto fails early with EINVAL.
Baseline (stock netgraph7 ng_base.c)
trigger.c does an A/B on ONE control socket:
- GOOD: 250,000 Γ 4148-byte sends to path "." (own node; responses drained). free pages 843954 β 843906 (Ξ 48 pages β noise). No leak. (run.log)
- BAD: sends to "df3474_no_such_node_xyz:" β each sendto returns ENOENT
(the exact
ng_path2nodereffailure that reachesng_address_path()'sNG_FREE_ITEM(item)at ng_base.c:3096).
Two independent runs (run.log = 250k target, run.2.log = 50k target):
- 3k-send smoke: free 847737 β 844942 (Ξ 2795 pages β 11 MB β 3000 Γ ~4 KB msg β matches per-send leak exactly, while the interleaved GOOD phase cost nothing).
- 50k-send run and 250k-send run both ended with the guest wedge-killed:
serial console shows
Warning: objcache(netgraph) exhausted on cpu0!(wedge_serial.log), ssh stopped answering,vm.sh statusβ down, no panic. Mechanism: ~4096 leaked items exhaust theng_ocobjcache permanently;objcache_get(ng_oc, M_WAITOK)then blocks in a non-interruptiblessleep("objcache_get")(kern_objcache.c:547-552) forever; the loop's messages leak kmem on top.
So: unbounded kernel memory leak + permanent non-interruptible wedge of netgraph-consuming threads = local DoS with root on a netgraph7 system (race-dependent variants exist for unprivileged data sockets, see README).
Fix validation
fix.diff (3 hunks, ng_address_hook/ng_address_path/ng_address_ID):
when item->refs == 0 (fresh, never-sent item β the only case whose
reference is never dropped by ng_apply_item()), call ng_free_item(item)
which frees the item plus attached msg/mbuf and drops embedded node/hook
references; otherwise keep the mark-only NG_FREE_ITEM() for in-flight
items (their final unref happens at ng_base.c:2109).
Applied in-guest to /usr/src (patch -p1, clean), netgraph.ko rebuilt:
- BAD 100,000 sends on patched module: all ENOENT, completed in seconds,
free pages 841147 β 840580 β 840571 after settle (Ξ 567 pages β 2.2 MB
steady-state objcache/magazine fill β versus ~420 MB expected on stock);
dmesg | grep -c exhaustedβ 0; guest healthy (fix_run.log). - GOOD 50,000 sends on patched module: ok=50000, no regression, guest healthy (uptime continues, load 0.04).
Baseline wedged twice; patched survived 2Γ the bad-send volume plus the good phase. fix_status: fixed.
Why this is not a false positive
- md5 of in-guest /usr/src/sys/netgraph7/netgraph/ng_base.c == workspace file (44667ee8cb1e5724235aa8ce9400abfd).
- Source proof: NG_FREE_ITEM sets only a flag (netgraph.h:816-820);
NGQF_FREE is never tested anywhere (rg over sys/);
ng_address_* return without dropping the item reference; fresh items are
born with refs==0 (bzero ctor + ng_alloc_item), and the only unconditional
ng_unref_item()sits at the end of ng_apply_item() which never runs for items that failed addressing before ng_snd_item(). - The kernel itself flagged the exhausted item cache on the console.
Exploit chain
Not a memory-corruption primitive; ceiling is the demonstrated permanent kernel wedge + unbounded kmem growth (dos). No uid=0 route exists for this class.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
fixedBaseline (stock netgraph7): 50k failing sends -> 'objcache(netgraph) exhausted' + permanent guest wedge (2 occurrences). Patched (fix.diff applied to in-guest /usr/src, module rebuilt): 100k failing sends complete in seconds, free-page delta ~567 pages (steady-state, vs ~420MB expected on stock), zero exhaustion warnings, good-path regression check 50k sends OK. The leak and the wedge are gone.
['findings/poc/DF-3474/fix_run.log', 'findings/poc/DF-3474/wedge_serial.log', 'findings/poc/DF-3474/fix.diff']
Confirmed kernel references
- sys/netgraph7/netgraph/ng_base.c:3062
- sys/netgraph7/netgraph/ng_base.c:3094
- sys/netgraph7/netgraph/ng_base.c:3117
- sys/netgraph7/netgraph/netgraph.h:816
- sys/netgraph7/netgraph2.h:77
- sys/netgraph7/netgraph/ng_base.c:2109
- sys/netgraph7/netgraph/ng_base.c:2772
- sys/netgraph7/socket/ng_socket.c:310
- sys/kern/kern_objcache.c:547
Detail
Exploit chain
root (or unpriv+graph-owner race) -> repeated failing AF_NETGRAPH control sendto (ENOENT from ng_address_path) -> each send leaks struct ng_item + kmalloc'd ng_mesg payload -> after ~4096 sends the netgraph objcache is permanently exhausted -> any thread allocating a netgraph item with M_WAITOK sleeps forever non-interruptibly (kern_objcache.c:547-552) -> system-wide wedge until reboot (demonstrated); continued sends additionally drain kmem unboundedly. Ceiling: permanent local DoS; no memory-corruption/uid=0 route.
Evidence (decisive lines)
['findings/poc/DF-3474/run.log β baseline A/B: GOOD 250k sends free 843954->843906 (no leak); BAD phase then wedged guest', 'findings/poc/DF-3474/run.2.log β second baseline: 3k smoke shows 2795-page (~11MB) drop matching 3000 x ~4KB leaked msgs; 50k bad-run wedged guest again', "findings/poc/DF-3474/wedge_serial.log β kernel console: 'Warning: objcache(netgraph) exhausted on cpu0!' then silence, ssh dead", 'findings/poc/DF-3474/fix_run.log β patched ng_base.c: 100k BAD sends complete, free 841147->840580 (~steady-state), dmesg exhaustion count 0; GOOD 50k sends ok', 'findings/poc/DF-3474/trigger.c β A/B trigger (NG_VERSION 8 for netgraph7)', 'findings/poc/DF-3474/fix.diff β 3-hunk fix, applied and validated in-guest']
PoC changes
Trigger initially used stock userland NG_VERSION (2, old sys/netgraph stack) which netgraph7 rejects with EINVAL; set header.version=8 explicitly. Also enlarged the sockaddr to a variable-length buffer for the 24-char bogus path, added per-phase errno histogram, per-iteration response draining (so the GOOD control phase never hits ENOBUFS backpressure), and argv-selectable phases/iteration counts. Guest needed the netgraph7 modules built from /usr/src (stock /boot/kernel modules are the OLD sys/netgraph build - verified via strings) and a fresh boot so the old stack's AF_NETGRAPH domain registration was absent.
Verified recommended fix
In ng_address_hook/ng_address_path/ng_address_ID error paths, free never-sent items via ng_free_item() when item->refs == 0 (keep mark-only NG_FREE_ITEM for in-flight items) - see fix.diff.
Verdict
netgraph7's NG_FREE_ITEM() only sets a vestigial NGQF_FREE flag (netgraph.h:816-820, flag never tested anywhere); the item is only really freed when the reference ng_snd_item()/ng_apply_item() manages is dropped at ng_base.c:2109. Items that fail addressing BEFORE being sent (fresh items, refs==0) are therefore leaked together with their entire control message at ng_base.c:3066/3096/3122. A root loop of sendto() on an AF_NETGRAPH control socket to a bogus path leaks ~4.2KB per call and, after ~4096 items, permanently exhausts the netgraph objcache (maxalloc=4096): objcache_get(M_WAITOK) then blocks forever in non-interruptible ssleep, wedging the machine (kernel prints 'Warning: objcache(netgraph) exhausted on cpu0!'). Reproduced twice (guest wedge-killed, vm.sh status down); control experiment on the same socket with a valid path shows zero growth. Fixed by freeing fresh (refs==0) items via ng_free_item() at the three ng_address_* error sites; patched module survived 100k failing sends with flat memory and no exhaustion warning. Deterministic trigger requires root (control socket is SYSCAP_RESTRICTEDROOT-gated, ng_socket.c:182); unprivileged race-dependent variants via data sockets exist (ngd_send -> NG_SEND_DATA_FLAGS -> ng_address_hook).
No comments yet.