β¬’ DragonFlyBSD Kernel Audit
← triage Β· dashboard
DF-2368

Key-size validation is no-op KASSERT invalid keyBits silently yields Nr=0 and runaway loop / OOB read

Summary

rijndaelKeySetupEnc validates keyBits ONLY via KASSERT which in production/release kernels (INVARIANTS undefined) expands to do{}while(0). With invalid keyBits none of three size branches taken function returns Nr=0 leaving rk[4..] uninitialized. rijndael_set_key stores Nr=0 without checking. rijndaelEncrypt computes r=Nr>>1=0 loop if(--r==0)break sets r=-1 never re-reaches 0 so loop runs indefinitely advancing rk+=8 each iteration past ek/dk buffers reading OOB kernel memory until faults unmapped page -> kernel hang/panic. All current in-tree callers pre-validate keyBits so not directly exploitable today but latent DoS landmine. [VERIFIED pass2: reproduced - setkey 0 panics at the sole KASSERT on INVARIANTS, on production kernels returns 0 with Nr=0 and partially-written schedule, encrypt then walks rk +32B per iteration into fatal trap 12 at rijndaelEncrypt+0x1d9 4128 bytes past the 488-byte ctx; all in-tree callers pre-validate so latent KPI landmine; layered validation fix-validated - invalid sizes rejected, 16/16 KATs bit-identical]

Discussion (0)

No comments yet.

PoC verification

Evidence pack

findings/poc/DF-2368 Β· 28 files
FileTypeDescriptionSize
README.md β€” 4.6 KB ↓ raw
VERDICT.md β€” 8.1 KB ↓ raw
rijndael_harness.c β€” 18.2 KB view raw
rijndael_twin_tables.h β€” 17.9 KB view raw
rijnh.c β€” 2.9 KB view raw
build.sh β€” 168 B view raw
run_kat.sh β€” 277 B view raw
run_setkey.sh β€” 497 B view raw
run_poison.sh β€” 907 B view raw
run_idx.sh β€” 2.6 KB view raw
build_baseline.log β€” 5.6 MB ↓ download
build_fixed.log β€” 5.6 MB ↓ download
install_baseline.log β€” 80.4 KB view raw
install_fixed.log β€” 80.4 KB view raw
kat_baseline_dmesg.txt β€” 1.4 KB view raw
panic.setkey0.txt β€” 1.3 KB view raw
panic.poison.txt β€” 1.1 KB view raw
panic.direct.txt β€” 945 B view raw
fixrun_dmesg.txt β€” 45.3 KB view raw
idx_dmesg.txt β€” 42.6 KB view raw
idx_run.out β€” 10.3 KB view raw
idx_analysis.txt β€” 659 B view raw
idx_analysis2.txt β€” 94 B view raw
fix.diff β€” 2.0 KB view raw
env.txt β€” 365 B view raw
ttables.txt β€” 345 B view raw
verdict.json β€” 4.0 KB view raw
manifest.json β€” 1.8 KB view raw

DF-2368 β€” rijndael key-size validation is a no-op KASSERT (Nr=0 -> runaway round loop / OOB read)

File: sys/crypto/rijndael/rijndael-alg-fst.c (+ wrapper sys/crypto/rijndael/rijndael-api.c)

Verdict (see VERDICT.md for the full narrative)

REPRODUCED on the stock INVARIANTS guest kernel (impact: panic / latent kernel DoS; confidence: certain). Both halves of the finding were demonstrated through the REAL kernel functions:

  1. panic: Invalid key size (0) β€” rijndael_set_key(bits=0) reaches the KASSERT at rijndael-alg-fst.c:738, the ONLY validation. On production (non-INVARIANTS) kernels this compiles to do {} while (0) and rijndaelKeySetupEnc returns 0 at line 814, so the wrapper stores ctx->Nr = 0 with a partially-written schedule (rk[0..7] written unconditionally, lines 740-743/762-763/784-785).
  2. The Nr=0 consequence: the round loop (r = Nr >> 1; for(;;){ ... rk += 8; if (--r == 0) break; ... }, lines 951-1007) never terminates (--r runs -1, -2, ...) and walks rk forward 32 bytes per iteration, reading far past ek/dk. Demonstrated: 4128 bytes past the end of the 488-byte rijndael_ctx, fatal kernel-mode read fault at rijndaelEncrypt+0x1d9.

NOT reachable by an unprivileged user today: every in-tree caller pre-validates the key size (see VERDICT.md reachability section). This is a latent landmine for any future/unprivileged-fed caller β€” hence the original Low severity, which this verification confirms.

Fix validated A/B (fix.diff): invalid sizes rejected with zeroed schedules, Nr=0 encrypts return immediately, and all 16 FIPS-197 / SP800-38A / OpenSSL-derived KATs remain bit-identical.

Rig

Kernel-compiled harness (runtime kldload GPFs on this guest) driving the REAL rijndael_set_key / rijndael_encrypt / rijndaelKeySetupEnc through sysctl kern.rijnh.test. Build/run below; all ops log with the RIJNH: tag (dmesg).

Build

On the guest (as root, from this directory):

./build.sh              # installs rig into /usr/src, nativekernel build + install
                        # (buildkernel fails: "You must buildworld before
                        #  buildkernel" on this snapshot; nativekernel is the
                        #  documented native-tools path, same obj tree)
reboot

Run

./run_kat.sh                    # control: 16/16 KAT PASS + twin validation
./run_setkey.sh 0               # DEMO A: panic "Invalid key size (0)"
./run_poison.sh poison          # DEMO B: fatal trap 12, OOB read walk
./run_poison.sh direct          # DEMO B via raw rijndaelEncrypt
./run_idx.sh                    # DF-2367 side-channel evidence (shared rig)

After a panic the guest sits in DDB: recover with vm.sh reset harness-kernel (snapshot taken right after the rig kernel was installed).

Fix validation: patch -p1 < fix.diff in /usr/src, rebuild, reboot, re-run the same commands β€” see VERDICT.md for the recorded A/B.

Expected output (stock kernel)

  • run_kat.sh: RIJNH: KAT summary 16 pass 0 fail of 16 vectors, twinval 600 vectors, 0 mismatches, guest stays up.
  • run_setkey.sh 0: kernel panic, ssh dies; serial console shows panic: Invalid key size (0). + backtrace rijndaelKeySetupEnc <- rijndael_set_key <- rijnh_run_setkey (panic.setkey0.txt).
  • run_poison.sh poison|direct: Fatal trap 12: page fault while in kernel mode, fault virtual address = 0x800475000, Stopped at rijndaelEncrypt+0x1d9: xorl -0x8(%rdi),%ecx (panic.poison.txt, panic.direct.txt).

Expected output (fixed kernel, fix.diff applied)

  • run_setkey.sh 0|1|127|257: no panic; ctx.Nr=0 ek[0]=00000000 ... dk[0]=00000000 (rejected, schedules zeroed).
  • run_poison.sh poison|direct: RIJNH: poison: SURVIVED Nr=0 encrypt; ct=00000000.
  • run_kat.sh: 16 pass 0 fail β€” bit-identical to baseline (fixrun_dmesg.txt vs kat_baseline_dmesg.txt).

Artifacts

  • rijndael_harness.c / rijndael_twin_tables.h β€” kernel rig (twin tables are a verbatim copy of the Te tables from rijndael-alg-fst.c lines 56-386,718-722)
  • rijnh.c β€” userland driver (mmaps the guard-page layout for DEMO B)
  • build.sh, run_*.sh β€” exact commands
  • build_baseline.log / build_fixed.log / install_*.log β€” full kernel build logs
  • kat_baseline_dmesg.txt β€” stock-kernel KAT + twinval control (16/16, 600/600)
  • panic.setkey0.txt / panic.poison.txt / panic.direct.txt β€” serial-console panics
  • idx_dmesg.txt / idx_run.out / idx_analysis*.txt β€” DF-2367 evidence (shared)
  • fixrun_dmesg.txt β€” fixed-kernel A/B log
  • fix.diff β€” git-apply-able fix (validated on guest; never applied to sys/ here)
  • env.txt, ttables.txt β€” guest environment, T-table kernel addresses
VERDICT.md
↓ download raw

DF-2368 VERDICT β€” no-op keyBits validation -> Nr=0 -> runaway round loop / OOB read

Status: REPRODUCED (impact: panic β€” kernel DoS landmine; confidence: certain) Rig: DragonFly 6.5-DEVELOPMENT x86_64, stock X86_64_GENERIC with options INVARIANTS (config line 56) and device crypto (line 299), QEMU/KVM, kernel-compiled sysctl harness (runtime kldload GPFs on this guest). Guest: DragonFly dfbsd 6.5-DEVELOPMENT #1/#2 ... x86_64 (env.txt).

What the finding claims and what was proven

Claim 1 β€” validation is KASSERT-only (CWE-617)

rijndaelKeySetupEnc (rijndael-alg-fst.c:734-815) validates keyBits only via KASSERT(keyBits == 128 || keyBits == 192 || keyBits == 256, ...) at lines 738-739. The public kernel KPI wrapper rijndael_set_key (rijndael-api.c:37-43) passes bits through with no check of its own.

REPRODUCED on the stock INVARIANTS kernel (kernel #1 + rig):

RIJNH: cmd="setkey 0"
RIJNH: setkey: calling real rijndael_set_key(bits=0) now (ctx pre-zeroed)
panic: Invalid key size (0).
rijndaelKeySetupEnc() at rijndaelKeySetupEnc+0x3ed
rijndaelKeySetupEnc() at rijndaelKeySetupEnc+0x3ed
rijndael_set_key() at rijndael_set_key+0x1f
rijnh_run_setkey() ... rijnh_sysctl() ...
Debugger("panic")                                    (panic.setkey0.txt)

The panic IS the assertion firing β€” proof that the assert is the only gate. On a production (INVARIANTS-free) build the same call path compiles the KASSERT to do { } while (0) and falls through: none of the three if (keyBits == ...) branches is taken (lines 744/764/786) and the function returns 0 at line 814. The wrapper stores ctx->Nr = 0. Note rk[0..7] are written unconditionally before the branches (lines 740-743, 762-763, 784-785), so the stored state is a partially-written schedule with Nr=0. (rijndaelKeySetupDec has the same KASSERT-only shape and inherits the fix through rijndaelKeySetupEnc.)

Claim 2 β€” Nr=0 makes the round loop run away and read OOB

rijndaelEncrypt (line 862) compiles the #else /* !FULL_UNROLL */ path (FULL_UNROLL is defined nowhere in the tree β€” verified): the loop

r = Nr >> 1;                       /* line 951: 0 when Nr==0 */
for (;;) {
    ... 4 lookups x 4 words ...    /* reads rk[4..7] */
    rk += 8;                       /* line 978: +32 bytes per iteration */
    if (--r == 0) break;           /* line 979: r was 0 -> -1, -2, ... */
    ... reads rk[0..3] ...
}

never breaks (--r starting at 0 goes negative; it would need 2^32-1 decrements to wrap to 0 again) and walks rk forward 32 bytes per iteration, past the 60-word ek/dk schedules.

REPRODUCED β€” the rig builds a REAL rijndael_ctx with a valid 128-bit rijndael_set_key at the end of a mapped (pre-faulted) user page followed by an unmapped page, forces ctx->Nr = 0 (byte-identical to the state the broken rijndael_set_key stores), and calls the real rijndael_encrypt (rijndael-api.c:53-57) β€” i.e. the full public API, not a synthetic call:

RIJNH: poison: ctx built (Nr was 10), ctx->Nr forced to 0; entering rijndael_encrypt(ctx) now
Fatal user address access from kernel mode from rijnh at ffffffff809f1c09
Fatal trap 12: page fault while in kernel mode
fault virtual address  = 0x800475000
fault code             = supervisor read data, page not present
Stopped at rijndaelEncrypt+0x1d9: xorl -0x8(%rdi),%ecx   (panic.poison.txt)

The faulting VA is 0x1200 (4608) bytes past ek β€” i.e. the loop executed ~144 iterations and read 4128 bytes past the end of the entire 488-byte ctx (DragonFly services kernel-mode demand faults on user VA while the surrounding mapping exists, so the walk crossed the ctx page and stopped at the first truly non-present page). In the real in-kernel scenario the ctx lives in kernel memory (wlan_ccmp ctx, cryptoapi session ctx), where the same walk would consume adjacent kernel objects and only trap β€” if ever β€” at an unmapped KVA hole up to gigabytes later: exactly the claimed kernel hang/panic. Identical result via a direct rijndaelEncrypt(ek, 0, ...) call (panic.direct.txt), confirming the runaway is in the core function, not the wrapper.

Control (not a rig bug)

16/16 KAT vectors (FIPS-197 App. C.1-C.3, SP800-38A ECB, and an OpenSSL-derived extended set) PASS through the same real functions with correct encrypt+decrypt round-trip (kat_baseline_dmesg.txt); the instrumented twin used for the DF-2367 evidence matches the real function byte-exact on 600 random vectors.

Reachability honesty (who can actually hit this)

Not reachable from unprivileged userspace today β€” every in-tree caller pre-validates the key size:

So the finding stands as filed: a latent DoS landmine at the kernel KPI boundary (one future caller that forgets to pre-validate turns invalid keyBits into a kernel hang/panic), Low severity, CWE-617. The guest harness reaches it only because the rig is compiled into the kernel for verification.

Privesc assessment (team directive: crypto-core class β€” close or flag)

CLOSED as non-privesc with reasoning: the runaway loop performs OOB reads only (round-key words are XOR-ed into on-stack cipher state; nothing OOB is written), the walked data never reaches userspace, and the terminal behavior is a fatal trap. No write primitive, no controlled pointer, no info return path β€” there is no escalation chain from this bug. Impact ceiling: kernel DoS.

Fix validation (A/B, same guest, same rig)

fix.diff (generated against the read-only sys/ tree; applied only inside the guest /usr/src with patch -p1; git apply --check clean on the host):

  1. rijndaelKeySetupEnc/rijndaelKeySetupDec: runtime check returning -1 for keyBits not in {128,192,256} (KASSERT kept for INVARIANTS builds).
  2. rijndael_set_key: rejects invalid bits β€” stores Nr = 0 and zeroes ek/dk (no stale schedule reuse) without calling the setup functions.
  3. rijndaelEncrypt/rijndaelDecrypt: guard Nr != 10 && != 12 && != 14 -> immediate return, killing the runaway even for direct callers.

Baseline (kernel #1) vs fixed (kernel #2):

input / op stock kernel fixed kernel
setkey 0/1/127/257 panic (assertion) no panic; Nr=0, ek/dk zeroed
setkey 128/192/256 Nr=10/12/14 Nr=10/12/14, identical schedules
poison (Nr=0 + encrypt) Fatal trap 12 in rijndaelEncrypt SURVIVED (guard returns), ct untouched
direct rijndaelEncrypt Fatal trap 12 SURVIVED
KAT 16 vectors 16/16 PASS 16/16 PASS, bit-identical ciphertexts
twinval 600/600 match 300/300 match

Full log: fixrun_dmesg.txt (fixed), kat_baseline_dmesg.txt (baseline). Bug inputs now rejected/clamped and all FIPS-197 vectors bit-identical β€” fix objective met.

Notes

  • The guest was returned to the pristine with-src snapshot after verification.
  • PoC sources: rijndael_harness.c (kernel rig), rijnh.c (userland driver); sha256 in verdict.json.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.

Fix verification

not_testable
↓ fix.diffper-fix-DF-2368

Confirmed kernel references

Detail

Evidence (decisive lines)

panic.setkey0.txt (panic: Invalid key size (0) with rijndael_set_key -> rijndaelKeySetupEnc backtrace); panic.poison.txt and panic.direct.txt (Fatal trap 12 page fault while in kernel mode, fault VA 0x800475000 = 4128 bytes past the ctx, Stopped at rijndaelEncrypt+0x1d9 xorl -0x8(%rdi),%ecx); kat_baseline_dmesg.txt (16/16 KAT control); fixrun_dmesg.txt (fixed kernel A/B: setkey 0/1/127/257 rejected, poison/direct SURVIVED, 16/16 KAT identical); fix.diff; build_baseline.log/build_fixed.log

PoC changes

Original filing had no runnable PoC (poc_path null). Rig authored from scratch: kernel-compiled sysctl harness because runtime kldload GPFs on this guest; make buildkernel fails on this snapshot without buildworld so the documented nativekernel path was used; three compile iterations were needed (Werror sequence-point on index logging, kernel __va_list instead of va_list).

Verified recommended fix

Runtime keyBits validation returning -1 in rijndaelKeySetupEnc/Dec plus rejection (Nr=0, zeroed schedules) in rijndael_set_key and an Nr sanity guard in rijndaelEncrypt/rijndaelDecrypt, exactly as in fix.diff

Verdict

Both halves reproduced on the stock INVARIANTS guest through the real kernel API via a kernel-compiled sysctl rig. (1) rijndael_set_key with keyBits outside 128/192/256 panics at the KASSERT in rijndaelKeySetupEnc (rijndael-alg-fst.c:738) - the only validation present; on production INVARIANTS-free kernels that assert compiles out and the function returns 0, so rijndael_set_key stores Nr=0 with a partially written schedule. (2) With the Nr=0 state the rijndaelEncrypt round loop (r = Nr>>1 = 0, break test on --r) never terminates and walked rk forward 32 bytes per iteration, reading 4128 bytes past the end of the 488-byte rijndael_ctx before a fatal kernel-mode read fault inside rijndaelEncrypt+0x1d9 - exactly the claimed OOB runaway. Not unprivileged-reachable today: all in-tree callers pre-validate key size (wlan_ccmp 16 bytes, cryptoapi 128/192/256, rijndael_makeKey), so this is a latent kernel-DoS landmine at the KPI boundary as filed (Low). Privesc closed: the loop performs OOB reads only, no write primitive, no data returned to userspace - no escalation chain. Fix validated A/B on the same guest: invalid sizes rejected with zeroed schedules, Nr=0 encrypts return immediately, and all 16 FIPS-197/SP800-38A KATs bit-identical.