DF-3531 — netgraph7 ng_tty: closing a tty with a pending chand callout

permanently leaks the ng_item + node reference

(ng_tty twin of DF-3515/DF-3523, same ng_uncallout defect)

What

ng7 ng_tty arms its output-drain timer with

sys/netgraph7/tty/ng_tty.c:453-454 (ngt_start)
    if (!IF_QEMPTY(&sc->outq) && !callout_pending(&sc->chand))
        ng_callout(&sc->chand, sc->node, NULL, 1, ngt_timeout, NULL, 0);

ng_callout() (sys/netgraph7/netgraph/ng_base.c:3248-3268) stores a struct ng_item — holding one NG_NODE_REF (ng_base.c:3257) — as the callout argument. Tearing the line discipline down runs

sys/netgraph7/tty/ng_tty.c:252-253 (ngt_close)
    if (callout_pending(&sc->chand))
        ng_uncallout(&sc->chand, sc->node);

ng_uncallout() (ng_base.c:3273-3296) frees that item only inside if ((rval > 0) && ...). On this kernel callout_stop() of an ARMED, not-yet-fired callout completes the stop inline and returns 0 (the DF-3515 _callout_cancel_or_stop contract violation, kern_timeout.c early return 0), so the gate never passes and the item is orphaned. Even when the gate passes, NG_FREE_ITEM() is flag-only in ng7 (netgraph.h:787-793) — ng_free_item() is never called. Every tty close with a pending chand callout permanently leaks:

• one struct ng_item (~141 B incl. objcache backing),
• one node reference — pinning the whole ng_node in the ID hash and pinning the module (kldunload ng_tty stays "Device busy").

Trigger

Root (ngt_open requires SYSCAP_NONET_NETGRAPH, ng_tty.c:188): open a pty, set ldisc NETGRAPHDISC(6), connect an ng7 data socket to "ttyN:hook", queue more output than the pty drains (master unread), and close the line discipline (fd close or TIOCSETD-away). The drain timer is then pending at ngt_close time and the cancel leaks.

Repeating this turns routine ng_tty use into unbounded permanent kernel memory growth; the accumulating leaked refs also push the ng7 node id/name hashes across the (separately filed, DF-3514) rehash panic threshold — observed here: the very first 20-cycle smoke run without the DF-3514 module fix wedged the guest in panic: hashdestroy: hash not empty.

Files

ngtty_leak.c   PoC (leak mode + "drain" control mode)
build.sh       build ng7 modules (with DF-3514 rehash fix only, so
               the known DF-3514 module-build panic does not mask
               the measurement) + compile the PoC
run.sh         baseline -> leak -> control -> leak -> kldunload check
fix.diff       two-layer fix validated here (kern_timeout.c contract
               + ng_base.c item free), applied to the guest for the
               fix-validation kernel/module

Build (in-guest, root)

cd /usr/src/sys/netgraph7/netgraph && make
cd ../socket && make
cd ../tty    && make
kldload .../netgraph.ko .../ng_socket.ko .../ng_tty.ko
cc -O2 -o ngtty_leak ngtty_leak.c

Run (in-guest, root)

vmstat -m | grep -E '^[[:space:]]+netgraph '   # baseline
./ngtty_leak 2000          # stall-close: LEAKS
./ngtty_leak 2000 drain    # drained control: ~flat
kldunload ng_tty           # -> "Device busy"

Observed (guest, DragonFly 6.5-DEVELOPMENT #0 INVARIANTS stock)

baseline                          12 allocs   1.69K
after 2000 stall-close cycles   1.96K allocs   283K   (+~1.9 allocs/cycle)
after 2000 drain-close cycles   2.32K allocs   334K   (mostly flat; the
                                     residual is cycles whose drain
                                     outlasted the sleep windows and hit
                                     the same leak path)
after 2000 more stall-cycles    3.98K allocs   573K   (monotone growth)
kldunload ng_tty                "Device busy" (refs=0, pinned by leaks)

With BOTH fix layers installed (kern_timeout.c PREVENTED-reporting + ng_base.c ng_free_item), the identical 2000-cycle stall-close run is FLAT — see VERDICT.md and fix_run.log.

IMPORTANT verification-stage discoveries (read VERDICT.md)

1. The baseline node-count growth is TWO stacked leaks: this one (items, via ng_uncallout) plus DF-3534 — ng7 ngt_shutdown never drops the constructor node reference, so every ng_tty node leaks (even a bare TIOCSETD on/off cycle with no hook, no data, no timer).
2. The ng_callout oitem-free hunk from DF-3514/DF-3523's fix.diff is UNSAFE for ng_tty's 1-tick self-rearming timer (re-arm from inside the callback frees the in-flight fired item → double-free; reproduced). It is excluded from this pack's fix.diff.
3. The layer-1 kern_timeout fix, as validated, is not dispatch-precise: under close-vs-fire racing it reported PREVENTED for an already- dispatched callback and ng_uncallout freed an in-flight item (delayed memory chunk already free panic — panic.txt). fix_status = inconclusive for that reason; the leak itself is flat.

DF-3531 — VERDICT

Finding: netgraph7 ng_tty — closing the tty while the chand drain callout is armed permanently leaks the timer ng_item and its node reference (the ng_tty twin of DF-3515/DF-3523's ng_uncallout defect).

Status: REPRODUCED (leak, certain) on the stock INVARIANTS guest. Fix: leak measurably eliminated by layers 1+2, but stress-testing the patched state surfaced a residual fire/close race that can free an in-flight timer item → heap double-free (delayed chunk_mark_free panic). fix_status = inconclusive — see "Fix validation" below.

Root cause (line-accurate)

• ngt_start() arms the pty-drain timer: sys/netgraph7/tty/ng_tty.c:453-454 ng_callout(&sc->chand, sc->node, NULL, 1, ngt_timeout, NULL, 0)
• ng_callout() (sys/netgraph7/netgraph/ng_base.c:3248-3268) allocates a struct ng_item carrying one NG_NODE_REF (ng_base.c:3257) and stores it as the callout argument.
• ngt_close() cancels it: sys/netgraph7/tty/ng_tty.c:252-253 if (callout_pending(&sc->chand)) ng_uncallout(&sc->chand, sc->node);
• ng_uncallout() (ng_base.c:3273-3296) frees the item only under if ((rval > 0) && callout_func(c) == trampoline && NGI_NODE(item) == node). On the stock kernel callout_stop() of an ARMED not-yet-fired callout completes inline and returns 0 (DF-3515: _callout_cancel_or_stop early return 0 before the PREVENTED-reporting path, sys/kern/kern_timeout.c) → the gate never passes → the never-sent item (refs==0) is orphaned forever: one ng_item + one node ref leaked per close. Even if the gate passed, NG_FREE_ITEM() is flag-only in ng7 (sys/netgraph7/netgraph.h:787-793) — ng_free_item() is never invoked.

Reproduction (baseline: stock kernel #0 + stock modules)

Harness: ngtty_leak.c (root; ngt_open requires SYSCAP_NONET_NETGRAPH, ng_tty.c:188). Per cycle: /dev/ptmx pty → TIOCSETD=NETGRAPHDISC(6) → NGIOCGINFO → ng7 NGM_CONNECT from a data-socket node to "ttyN:hook" → sendto 1500 B (master never read → t_outq pinned at NGT_HIWATER=400, remainder sits in sc->outq, chand re-arms every tick) → TIOCSETD back to termios (the tty.c:1137 l_close path calls ngt_close synchronously; same ngt_close code as fd-close) → sockets closed.

vmstat -m | grep netgraph:
baseline                         12 allocs   1.69K
2000 stall-close cycles        1.96K allocs   283K     <-- ~1/cycle leaked
2000 drain-close (control)     2.32K allocs   334K     (mostly flat; the
                                  residual = cycles whose drain outlasted
                                  the sleep windows and hit the same leak)
2000 more stall-close cycles   3.98K allocs   573K     (monotone)
kldunload ng_tty               "Device busy" (refs=0 — pinned by leaked refs)

Corroboration: the very first 20-cycle smoke run (stock modules, before the known DF-3514 module-build rehash fix was pre-applied) wedged the guest in panic: hashdestroy: hash not empty — the leaked node refs had already pushed ng_nodes past the rehash threshold within ~20 cycles (DF-3514 feed).

Note on measurement: the node-count part of the baseline delta is contaminated by a second, independent ng_tty leak discovered during this verification (DF-3534: ngt_shutdown never drops the constructor node ref — every ng_tty node leaks regardless of the callout). The item leak (+1.95K/2000, M_NETGRAPH netgraph type) is DF-3531's alone: with DF-3534's node fix but stock ng_uncallout behavior … items still climb; with the ng_uncallout fix items go flat while nodes still climb (fix_run.3.log) — the two leaks are cleanly separable and each fix is independently effective for its own allocation class.

Fix validation

Layers applied (fix.diff): 1. sys/kern/kern_timeout.c — _callout_cancel_or_stop reports PREVENTED for inline-completed stops (the DF-3515 contract fix, same diff DF-3523 validated). 2. sys/netgraph7/netgraph/ng_base.c — ng_uncallout() calls ng_free_item(item) after the flag-only NG_FREE_ITEM (layer 2).

Result (kernel #1 + rebuilt modules, fix_run.final2.log):

baseline:  netgraph(items)=2 in-use, netgraph_node=2
nodd 500 (ldisc on/off only):        nodes 2 -> 2 (flat; DF-3534 effect)
stall-close 2000:                    items 9 in-use (2.45K allocs cycled),
                                     nodes 2 (flat)
stall-close 2000 (2nd):              items 9, nodes 2  (FLAT over 4000)
hooks/paths/socks in-use 0 throughout; kldunload ng_tty succeeded once
the ref pinning was gone (final battery, earlier run: KLDUNLOAD_OK).

Residual hazard (why fix_status=inconclusive): under aggressive close-vs-fire racing (drain-mode stress, and earlier an instrumented build) the patched state double-freed an in-flight timer item: * instrumented module: Fatal trap 12 in ng_snd_item (lock addl $0x1,0x80(%rdi), el_node==NULL) — an item already dispatched to the cpu0 netgraph port had been freed by ng_uncallout; * clean modules: delayed panic: memory chunk 0x... is already free! (chunk_mark_free via slab_cleanup/slotimer_callback) after the flat 4000-cycle battery, during drain-mode stress.

Mechanism: when ngt_close's ng_uncallout races the 1-tick fire — the trampoline has dequeued the item and handed it to lwkt_sendmsg (cpu0) — the layer-1 patched callout_stop can report PREVENTED (rval>0) for a callout whose callback already dispatched, so ng_uncallout frees an item that is queued/in-flight (refs==1 held by ng_snd_item). Stock code cannot do this (it never frees). CONSEQUENCE FOR UPSTREAM: the DF-3515/DF-3523 kern_timeout fix layer must guarantee PREVENTED is reported only when the callback never dispatched; alternatively ng_uncallout must distinguish "stopped before dispatch" from "already dispatched" (e.g. trampoline-side arg clearance). This also means the ng_callout oitem-free hunk shipped in DF-3514/DF-3523's fix.diff is dangerous for self-rearming 1-tick timers (ng_tty re-arms from inside ngt_timeout): it frees the in-flight fired item on every re-arm — do NOT include that hunk for ng_tty-class users (it was excluded from this pack's fix.diff after it reproduced the same double-free signature immediately).

What was NOT the leak (negative results)

• v1 sys/netgraph/tty/ng_tty.c uses plain callout_reset/callout_stop with no ng_item — no ng_uncallout twin exists in v1.
• The generic ng7 socket/connect/data path leaks nothing (socksock control flat) — the tty ldisc is the sole leaker of its own items/nodes.

Environment

Baseline kernel: DragonFly 6.5-DEVELOPMENT #0 (stock INVARIANTS X86_64_GENERIC, Thu Jul 2 06:02:54 2026). Fix kernel: #1 Mon Sep 14 09:24:11 2026 (kern_timeout layer-1). Modules built in-guest from /usr/src/sys/netgraph7 with gcc80.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.