β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-3271

netgraph7 ng_ID_rehash panics INVARIANTS kernels under ordinary node churn (hashdestroy: hash not empty); ng_base.c:996 also passes the wrong mask to hashdestroy (latent OOB read)

Summary

Out-of-band discovery during DF-0638 verification churn stress-testing: repetitive, completely ordinary netgraph7 node creation (socket node + mkpeer deflate nodes - routine ngctl/MPD graph setup) panics INVARIANTS kernels: 'panic: hashdestroy: hash not empty' from ng_ID_rehash inlined into ng_make_node_common (:658 -> :973-999 -> hashdestroy KASSERT). REPRODUCED 4x including on a fresh guest (12-iteration mkpeer loop, 8 nodes/iteration; trace hashdestroy <- ng_make_node_common+0x21f <- ng_make_node <- ng_apply_item <- ngthread, guest dead in DDB). ng_base.c:996 passes ng_name_hmask instead of ng_ID_hmask to hashdestroy (copy-paste from ng_name_rehash:964) - objectively a wrong-variable defect that OOB-reads LIST heads past the old ID-hash allocation when the name mask exceeds the ID mask. HONEST fix_failed: a guest-built one-line mask fix did NOT stop the panic (frame-offset shift proves the patched module was active), so the immediate cause is deeper nd_idnodes list-state corruption surviving LIST_FOREACH_MUTABLE+LIST_REMOVE - needs upstream ddb-level debugging. Privileged local user (AF_NETGRAPH root-only); non-INVARIANTS kernels compile out the KASSERT - if the list corruption is real it may silently corrupt the ID hash (unresolved). [RESOLVED by DF-3514: the rehash migration loops unlink only under #ifdef INVARIANTS (double-insert on module builds) - the mask bug was necessary-but-insufficient; DF-3514 validated fix supersedes]

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-3271 Β· 4 files
FileTypeDescriptionSize
panic.txt panic-traces 4 occurrences incl. fresh-guest baseline and patched-module persistence 2.4 KB view raw
repro.sh run-script node-churn loop (needs df3270_exploit from DF-3270 pack) 724 B view raw
fix.diff fix one-line ng_name_hmask->ng_ID_hmask correction at ng_base.c:996 - real latent defect but INSUFFICIENT for the panic (proven) 271 B view raw
VERDICT.md verdict analysis incl. disproven hypothesis and double-link corruption hypothesis 4.3 KB ↓ raw
VERDICT.md verdict analysis incl. disproven hypothesis and double-link corruption hypothesis
↓ download raw

DF-3271 β€” netgraph7 ng_ID_rehash/hashdestroy panic under node churn

What happens

Repetitive, completely ordinary netgraph7 node creation (socket node + mkpeer deflate nodes, i.e. what any ngctl script or MPD does when graphs are set up and torn down) panics INVARIANTS kernels:

panic: hashdestroy: hash not empty
hashdestroy() at hashdestroy+0x3a
hashdestroy() at hashdestroy+0x3a            <- inlined ng_ID_rehash
ng_make_node_common() at ng_make_node_common+0x21f
ng_make_node() at ng_make_node+0x3b
ng_apply_item() at ng_apply_item+0x666
ngthread() at ngthread+0x18

Guest dies in DDB; ssh gone; vm.sh status => down. Reproduced 4 times (panic.txt), including on a FRESH guest with only the netgraph7 module stack loaded (occurrence 3).

Where

sys/netgraph7/netgraph/ng_base.c:

  • ng_make_node_common() (:618) increments ng_nodes and calls ng_ID_rehash() when ng_nodes * 2 > ng_ID_hmask (:656-658).
  • ng_ID_rehash() (:973-999) moves every node from the old ng_ID_hash into a doubled table (under #ifdef INVARIANTS it LIST_REMOVEs each node first, :986-994) and then calls hashdestroy(ng_ID_hash, ...) (:996).
  • hashdestroy() (sys/kern/kern_subr.c:348) KASSERTs every bucket in 0..hashmask is empty before kfree.

The KASSERT trips, i.e. at hashdestroy time the old table still looks non-empty although the rehash loop just LIST_REMOVEd everything β€” consistent with a corrupted/double-linked nd_idnodes list state that makes LIST_FOREACH_MUTABLE + LIST_REMOVE leave a stale bucket head.

What is NOT the (sole) cause β€” proven experimentally

ng_base.c:996 passes the wrong mask to hashdestroy:

hashdestroy(ng_ID_hash, M_NETGRAPH_NODE, ng_name_hmask);   /* should be ng_ID_hmask */

That is objectively a defect: when ng_name_hmask > ng_ID_hmask (old), hashdestroy reads LIST heads past the end of the old ID-hash allocation (kernel OOB read, up to (name_hmask - ID_hmask)+1 buckets = e.g. 256+ bytes) and KASSERTs on garbage; the sibling ng_name_rehash() (:964) passes the correct mask, confirming the copy-paste origin. The one-line fix (ng_name_hmask -> ng_ID_hmask, see fix.diff) was built and tested under the identical repro loop β€” the panic persisted (occurrence 4 in panic.txt, with the patched module proven active by the shifted ng_make_node_common+0x21b frame offset). So the panic has a deeper cause in the rehash/list bookkeeping that requires ddb-level debugging of the live hash state; the wrong mask at :996 remains a real latent OOB-read defect that should be fixed regardless.

Impact

  • INVARIANTS/debug kernels: reliable kernel panic (local DoS) from privileged netgraph node churn (AF_NETGRAPH socket creation is root-only on this system). All 4 occurrences deterministic within a 12-iteration loop.
  • Production (non-INVARIANTS) kernels: the KASSERT is compiled out; the observable symptom disappears, but if the underlying list-state corruption is real it may silently corrupt the ID hash (unresolved; needs upstream debugging). The :996 wrong-mask direction only reads/KASSERTs under INVARIANTS, so by itself it is debug-kernel-only.

Reproduce

See repro.sh (needs the DF-3270 client binary df3270_exploit, whose -g 6 mode performs the churn loop). Guest must have the netgraph7 modules built (see DF-3270 README). Expected on a stock INVARIANTS guest: panic within <= 12 iterations (observed: iteration 2-3 once prior netgraph activity exists, fresh guest also panics within the loop).

Fix attempt (fix.diff) β€” INSUFFICIENT

- hashdestroy(ng_ID_hash, M_NETGRAPH_NODE, ng_name_hmask);
+ hashdestroy(ng_ID_hash, M_NETGRAPH_NODE, ng_ID_hmask);

Built, loaded, re-tested: panic persists (occurrence 4). Recorded as fix_status=fix_failed. Upstream needs to instrument ng_ID_rehash (dump old bucket heads at destroy) to pin the list corruption β€” the double-linked-list hypothesis (node linked into two buckets at rehash time) is the most consistent with the KASSERT firing with the correct mask.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

fix_failed
baseline reproduced→ patch + rebuild →patched reproduced

One-line ng_name_hmask->ng_ID_hmask fix built and loaded; identical repro loop still panics (occurrence 4 in panic.txt) - the wrong mask is a real latent OOB-read defect but NOT the panic cause; root cause lies deeper in the ID-hash rehash/list bookkeeping and needs upstream debugging.

findings/poc/DF-3271/panic.txt (occurrence 4, frame offset +0x21b proves patched module active); findings/poc/DF-3271/fix.diff
↓ fix.diffDragonFly dfbsd 6.5-DEVELOPMENT #0 (same kernel; netgraph.ko module rebuild+reload with the one-line fix)

Confirmed kernel references

Detail

Evidence (decisive lines)

['panic.txt: 4 traces, identical signature, incl. fresh-guest occurrence and patched-module occurrence (ng_make_node_common+0x21b vs stock +0x21f)', 'repro.sh: 12-iteration churn loop; guest dies in DDB (ssh gone, vm.sh status=down)']

PoC changes

no seed; discovered during DF-3270 churn stress; repro loop reuses the DF-3270 client in -g 6 mode (8 nodes/iteration)

Verified recommended fix

fix the wrong mask at ng_base.c:996 (ng_name_hmask -> ng_ID_hmask) AND instrument/debug the rehash list state: dump old bucket heads at hashdestroy to pin why a bucket survives LIST_FOREACH_MUTABLE+LIST_REMOVE; suspect nd_idnodes double-linkage across rehash/death interleaving

Verdict

REPRODUCED 4x (2 organic during DF-3270 work, 1 fresh-guest baseline loop, 1 with a patched netgraph.ko). Ordinary netgraph7 node-creation churn (socket node + mkpeer deflate nodes, i.e. routine ngctl/MPD graph setup) panics INVARIANTS kernels: 'hashdestroy: hash not empty' from ng_ID_rehash inlined into ng_make_node_common (ng_base.c:658 -> :973-:999 -> hashdestroy, kern_subr.c:348 KASSERT). The old ID-hash table still contains (or appears to contain) entries at destroy time although the rehash loop LIST_REMOVEs everything - consistent with nd_idnodes list-state corruption (double-link hypothesis in VERDICT.md). The obvious wrong-variable defect at ng_base.c:996 (hashdestroy(ng_ID_hash,...,ng_name_hmask) - should be ng_ID_hmask; copy-paste from ng_name_rehash; an OOB read of LIST heads when name mask exceeds ID mask) was fixed and tested: panic PERSISTS with the patched module (frame offset shift proves it was active) - so it is a real latent defect but NOT the panic's cause; the deeper list-corruption needs upstream ddb-level debugging.