ng7 ng_cisco: disconnecting the downstream hook with the armed keepalive ng_callout leaks the timer ng_item + node reference (DF-3531's ng_tty twin)
Summary
cisco_newhook('downstream') arms the 10s keepalive via ng_callout, which allocates a struct ng_item holding one NG_NODE_REF. Every teardown of that hook runs ng_uncallout, which frees the pending item only when callout_stop() returns >0 - but on this kernel an armed, not-yet-fired cancel completes inline and returns 0 (DF-3515 kern_timeout contract violation), so the gate never passes: one ng_item + one node reference is orphaned per armed-cancel. The leaked reference pins the ng_node in the ID hash forever and pins the module; repeating connect/disconnect grows the kernel heap without bound, and the pinned zombie nodes push ng_nodes past the ng7 ID-hash rehash threshold, panicking stock kernels via DF-3514's hashdestroy. VERIFIED on the guest (ng7 modules built from pristine /usr/src): leak runs +300 allocs/300 cycles in BOTH zones (netgraph 302->602, netgraph_node in lockstep, 84.7K/100K), inet-hook control run FLAT, second leak run +300 again (monotone); kldunload ng_cisco refused 'Device busy'; a 3000-cycle run on stock modules panicked mid-run 'panic: hashdestroy: hash not empty' in ng_make_node_common (panic.txt). Framework fix (same defect as DF-3531, not node-local): kern_timeout dispatch-precise PREVENTED layer + ng_uncallout must actually ng_free_item().
Discussion (0)
PoC verification
Evidence pack
findings/poc/DF-3536 Β· 11 files| File | Type | Description | Size | |
|---|---|---|---|---|
| ngcisco7_leak.c | β | 5.1 KB | view raw | |
| build.sh | β | 143 B | view raw | |
| run.sh | β | 90 B | view raw | |
| build.log | β | 3.0 KB | view raw | |
| run.log | β | 641 B | view raw | |
| panic.txt | β | 573 B | view raw | |
| env.txt | β | 936 B | view raw | |
| rehash_fix.diff | β | 841 B | view raw | |
| fix.diff | β | 3.3 KB | view raw | |
| VERDICT.md | β | 5.3 KB | β raw | |
| verdict.json | β | 3.4 KB | view raw |
DF-3536 β VERDICT
Finding: netgraph7 ng_cisco β disconnecting the downstream hook while
the 10-second keepalive ng_callout is armed permanently leaks the timer
ng_item and the node reference it holds. This is the ng_cisco twin of
DF-3531 (ng_tty), same framework defect, different reachable node type.
Status: REPRODUCED (leak, certain) on the stock INVARIANTS guest with
ng7 modules built from the pristine /usr/src snapshot.
Root cause (line-accurate)
cisco_newhook("downstream")arms the keepalive timer:sys/netgraph7/cisco/ng_cisco.c:218-220c /* Start keepalives */ ng_callout(&sc->handle, node, NULL, (hz * KEEPALIVE_SECS), &cisco_keepalive, (void *)sc, 0);ng_callout()allocates astruct ng_itemcarrying oneNG_NODE_REF(sys/netgraph7/netgraph/ng_base.c:3253-3258) and stores it as the callout argument (ng_base.c:3266-3267).- When the last hook goes away β normal disconnect,
NGM_RMHOOK,NGM_SHUTDOWN, peer teardown βcisco_disconnectcancels:sys/netgraph7/cisco/ng_cisco.c:416-418c if (pep->af == 0xffff) /* If it is the downstream hook, stop the timers */ ng_uncallout(&sc->handle, NG_HOOK_NODE(hook)); ng_uncallout()frees the pending item only whencallout_stop()returns0 (
sys/netgraph7/netgraph/ng_base.c:3281-3292):c rval = callout_stop(c); item = callout_arg(c); if ((rval > 0) && (callout_func(c) == &ng_callout_trampoline) && (NGI_NODE(item) == node)) { NG_FREE_ITEM(item); /* flag-only in ng7; frees nothing */ }- On this kernel,
callout_stop()of an ARMED, not-yet-fired callout completes the cancellation inline but returns 0 β the DF-3515callout_stop(9)contract violation (_callout_cancel_or_stop()earlyreturn 0before the PREVENTED-reporting path,sys/kern/kern_timeout.c:886-890). The gate at ng_base.c:3284 never passes, so the never-sent item (whose queue refcount is 0) is orphaned: oneng_item+ oneNG_NODE_REFleaked per armed-cancel. (Even if the gate passed,NG_FREE_ITEMis flag-only in ng7 andng_free_item()is never called β DF-3531 layer-2.)
Consequences per leaked reference: the ng_node struct is pinned forever in
the ID hash (ng_unref_node never runs), the type refcount never drops, and
kldunload ng_cisco is permanently refused ("Device busy"). Repeating
connect/disconnect grows the kernel heap without bound; the pinned zombie
nodes also push ng_nodes past the ng7 ID-hash rehash threshold, which on
stock modules feeds the DF-3514 hashdestroy panic (observed, panic.txt).
Reproduction
Measured runs: 300-cycle phases (harness ngcisco7_leak.c), ng7
netgraph.ko built with the DF-3514 rehash fix pre-applied (the identical
procedure DF-3531 used for its measured runs, so the separately-filed rehash
defect does not step on the leak differential):
| phase | M_NETGRAPH allocs | M_NETGRAPH_NODE allocs |
|---|---|---|
| 300 downstream (leak) cycles | 302 / 42.5K | (in lockstep, see env.txt) |
| 300 inet (control) cycles | 302 / 42.5K (FLAT) | β |
| 300 downstream cycles again | 602 / 84.7K (+300, linear) | 602 / 100K |
kldunload ng_cisco β can't unload file: Device busy (refs pinned).
Stock-module corroboration: the first full run (3000 downstream cycles, no
rehash pre-fix) panicked the guest mid-run:
panic: hashdestroy: hash not empty β ng_make_node_common β ng_make_node
β ng_apply_item β ngthread (panic.txt, serial console). The leaked
node references kept zombie nodes in the old ID hash until the rehash
threshold tripped β the same DF-3514 feed chain DF-3531 documented for
ng_tty.
Differential validity: the control run differs from the leak run ONLY in the
peer hook name (inet vs downstream) β i.e. only in whether the keepalive
ng_callout is armed and then cancelled by ng_uncallout. Node creation,
hook add/destroy and node teardown are byte-identical code paths.
Exploit chain
Not a corruption primitive: unbounded kernel-memory exhaustion (DoS) +
permanent module-unload denial, triggerable by any principal holding a
netgraph control socket (SYSCAP_RESTRICTEDROOT, ng_socket.c:182). Two
zones grow linearly with attacker-chosen cycle count.
Fix validation
fix_status = not_testable. The defect lives in framework code shared by
every ng7 timer user: the correct fix is the two-layer patch already
authored and stress-tested by DF-3531's evidence pack (layer 1:
kern_timeout.c must report PREVENTED for inline-completed cancels β a
kernel rebuild; layer 2: ng_uncallout must actually ng_free_item()).
DF-3531's pack validated that combination on this same guest and documented
its residual caveat (the layer-1 return must be dispatch-precise or a
close-vs-fire race double-frees an in-flight item). Rebuilding the kernel a
second time to re-run an already-validated fix adds no information; the
caveat is upstream's to resolve. fix.diff in this pack mirrors DF-3531's
two-layer fix.
Fix status (batch validation)
Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.
Fix verification
not_testableFramework-level two-layer fix (kern_timeout.c PREVENTED reporting + ng_uncallout ng_free_item) already authored and stress-validated by DF-3531's pack on this same guest, including its documented dispatch-precision caveat; re-validating requires a kernel rebuild and would only repeat DF-3531's experiment.
fix.diff mirrors DF-3531/fix.diff; see findings/poc/DF-3531/VERDICT.md 'Fix validation' for the kernel-rebuild results and the double-free caveat under close-vs-fire racing.
Confirmed kernel references
Detail
Exploit chain
root netgraph user repeats NGM_MKPEER(cisco,downstream)+NGM_RMHOOK cycles: +1 pinned ng_node and +1 leaked ng_item per cycle, unbounded kernel heap growth, permanent module-unload denial, and on stock modules a deterministic kernel panic once ng_nodes crosses the ID-hash rehash threshold (DF-3514 feed).
Evidence (decisive lines)
run.log (302 -> flat control -> 602 allocs, kldunload Device busy), env.txt (netgraph_node 602/100K in lockstep), panic.txt (stock-module hashdestroy panic), VERDICT.md root-cause with path:line
PoC changes
wrote harness from scratch modeled on DF-3531's ABI header: fixed sockaddr_ng family type (DragonFly sa_family_t is __uint8_t, not u_short β first build sent skewed paths and failed with ENOENT/ENXIO); ng7 modules for measured runs built with the DF-3514 rehash hunks pre-applied (rehash_fix.diff) so the separately-filed rehash defect does not mask the leak differential.
Verified recommended fix
ng_uncallout must free the orphaned timer item (ng_free_item) whenever the cancel completed, gated on a dispatch-precise PREVENTED report from callout_stop (kern_timeout.c early-return fix); see DF-3531.
Verdict
REPRODUCED (leak, certain): every ng7 cisco downstream-hook disconnect with the armed 10s keepalive ng_callout leaks one ng_item + one node reference because ng_uncallout (ng_base.c:3281-3292) frees the item only when callout_stop() > 0 and this kernel's callout_stop returns 0 for inline-completed armed cancels (kern_timeout.c:886-890, DF-3515). 300-cycle leak runs grow M_NETGRAPH/M_NETGRAPH_NODE by exactly +300 allocs each (linear, monotone); the inet-hook control run is flat; kldunload ng_cisco is permanently refused (Device busy); a 3000-cycle run on stock modules additionally panicked the guest via the DF-3514 rehash feed (panic: hashdestroy: hash not empty in ng_make_node_common). Trigger requires a netgraph control socket (SYSCAP_RESTRICTEDROOT).
No comments yet.