DF-3562 — serpent_set_key() unsigned key-length wraparound (pass-2, batch anchor pack)

This pack is the evidence anchor for the GLM-5.3 pass-2 audit of four DragonFlyBSD kernel crypto files:

Build

On the DragonFly guest (sources at /usr/src/sys, SHA256-verified identical to the audit tree — see env.txt era logs):

cd /tmp/DF-3562
./build.sh          # stages verbatim copies of the kernel sources; only
                    # kernel-only header includes are redirected to shims
                    # (recorded in verification/chachapoly/prep.sh)
./run.sh            # runs all four harnesses; exit 0 == all green

Prereqs: cc (guest base gcc 8.3), the pack directory.

What each part proves

• verification/serpent/serptest — 4 Linux-mainline testmgr KATs + 169 vectors from Botan's verified serpent corpus (128/192/256-bit keys, multi-block up to 1 KB) + 6000 random round-trips + 1000x iterated chains: 6349 checks, 0 fails.
• verification/serpent/serpoob — the DF-3562 OOB demonstrations (3/3 fault on pristine code, 0/3 on fixed code):
• Case A key_len=-64: lk = (u_int32_t)((-64+31)/32) = 0xFFFFFFFF → runaway ctx->l_key[i] fill loop (linear OOB write of key-buffer words, SIGSEGV at the first unmapped page).
• Case B key_len=-32: i = (u_int32_t)(-32/32) = 0xFFFFFFFF → single wild access at l_key[0xFFFFFFFF] (+16 GiB, SIGBUS).
• Case C key_len=129: (129+31)/32 = 5 words read from a 17-byte key buffer → 3-byte OOB read (SIGSEGV on a guard page).
• verification/twofish/twotest — 6 published KATs (official ecb_ival-derived + LibTomCrypt) + 1000x iterated + 15000 random round-trips: 15018 checks, 0 fails.
• verification/rijndael-api/rijnapitest — FIPS-197 ECB KATs, RFC 3602 CBC cases #1-#4, 192 pad round-trips, CFB1 single-block round-trip, and live confirmation records for the KNOWN pass-1 findings DF-0968 (CBC numBlocks==0 writes a full block past the advertised 8-byte payload — canary clobbered) and DF-0969 (ECB padDecrypt rejects valid padLen==16 on 6/6 exact-multiple inputs): 208 checks, 0 fails (known-bug confirmations counted separately).
• verification/chachapoly/cptest — RFC 8439 §2.8.2 AEAD (ct+tag byte-exact), RFC 8439 A.5 decrypt+re-encrypt, draft-irtf-cfrg-xchacha A.1 both directions, tamper rejection (ct/tag/aad), 264 boundary round-trips (0..4096 B × ad lengths × chacha/xchacha × separate and in-place), and the mbuf API through bounds-aborting DFly-faithful mbuf shims (9 lengths × 6 fragmentation patterns byte-matched against the contiguous result, corruption → EBADMSG): 22 checks, 0 fails.
• wg_e2e.log — live kernel end-to-end: two wg(4) peers over loopback UDP (kldload if_wg), 200/200 + 20/20 (1300 B) pings, 0% loss — the real in-kernel chacha20poly1305 mbuf/handshake paths.

Fix validation (DF-3562)

fix.diff adds the missing lower-bound / byte-multiple check to serpent_set_key(). fix_build.log / fix_run.log show it applied to a verbatim /usr/src copy of the file, rebuilt, and re-run: baseline 3/3 faults → patched 0/3 faults, KAT corpus still 6349/0. A full kernel rebuild was not performed for this fix: the changed line is unreachable from any in-tree caller (cryptoapi.c serpent_cbc_setkey / serpent_xts_setkey validate 128/192/256 (and 256/512 halved) bits before calling — see VERDICT.md), so no kernel code path can observe a behavior difference; the harness rebuild exercises the identical function source.

Files

build.sh run.sh                  exact build/run entry points
serpoob.c                        DF-3562 OOB demonstration
fix.diff                         the fix (git-apply-able vs sys/)
build.log run.log env.txt        full untrimmed guest logs
fix_build.log fix_run.log        fix-validation logs
wg_e2e.log                       live wg(4) end-to-end
verification/serpent/            serptest.c serpoob (built) kats_serpent.txt
verification/twofish/twotest.c
verification/rijndael-api/rijnapitest.c
verification/chachapoly/         cptest.c shim_systm.h shim_mbuf.h prep.sh

DF-3562 VERDICT — serpent_set_key() unsigned key-length wraparound; batch anchor for the 4-file crypto pass 2

Finding (sys/crypto/serpent/serpent.c)

serpent_set_key() validates only the upper bound of key_len (if (key_len > 256) return;, serpent.c:456). i and lk are u_int32_t (serpent.c:453), so negative or non-byte key_len values wrap/round catastrophically:

• key_len ≤ −63 — lk = (key_len + 31) / 32 truncates to a negative int (e.g. −64 → −1) which wraps to ~0xFFFFFFFF when compared against the unsigned loop counter: the load loop (while (i < lk) ctx->l_key[i] = htole32(*(u32*)(in_key + 4*i));, serpent.c:461-464) runs away — a linear OOB write of key-buffer words into ctx->l_key[i] for billions of words (~16 GiB), reading the caller's key buffer OOB the whole way. Demonstrated live (run.log: SIGSEGV, si_code=2, at the first unmapped page below the frame).
• key_len ∈ [−63, −32] (with (key_len+31)/32 == 0) — the load loop is skipped but i = key_len / 32 = −1/−2 → (u_int32_t)0xFFFFFFFF/ 0xFFFFFFFE → single wild read-modify-write at ctx->l_key[0xFFFFFFFF] (offset +16 GiB). Demonstrated live (SIGBUS, si_code=3). (key_len ∈ [−31, −1] additionally evaluates 1 << (key_len % 32) with a negative shift count — UB, benignly masked on x86.)
• key_len not a multiple of 8 (e.g. 129) — lk = (129+31)/32 = 5 reads 20 bytes from a 17-byte key buffer: up to 3 bytes OOB read. Demonstrated live (SIGSEGV on a guard page).
• key_len > 256 — silent early return leaves the caller's ctx untouched (cryptoapi pre-fills it with karc4random_buf, so no info leak there) and the session silently encrypts with a random key.

Reachability (why this is Low / defense-in-depth)

serpent_set_key has exactly one in-tree caller chain: cryptoapi.c serpent_cbc_setkey (cryptoapi.c:826-839) and serpent_xts_setkey (cryptoapi.c:927-943) — both switch on keylen_in_bytes * 8 and only call through for 128/192/256 (XTS: 256/512 halved to 128/256). cryptoapi_cipher_setkey is reached from hammer2/dm-crypt-style consumers with operator-supplied keys (root configured). No unprivileged or remote path exists today; the bug is the serpent twin of the twofish DF-0965/DF-2371 class (which pass 1 filed with the same reachability caveat). Filed so the primitive is fixed at the source, not at one caller.

Evidence

• run.log — serpoob section: 3/3 fault demonstrations with per-child si_code/si_addr.
• fix.diff, fix_build.log, fix_run.log — two-line fix (key_len < 0 || key_len > 256 || (key_len & 7)) applied to a verbatim /usr/src copy: 3/3 → 0/3 faults, KAT corpus still 6349/0. git apply --check passes against the audit tree. No kernel rebuild was performed: the only in-tree caller pre-validates the argument, so no kernel code path can reach the changed line; the harness compiles the identical function source.

Correctness verification (all four files, live on the guest)

Guest: DragonFly 6.5-DEVELOPMENT #0 (INVARIANTS) x86_64, gcc 8.3, guest /usr/src/sys SHA256-matched to the audit tree for all staged files.

serpent.c — 6349 checks, 0 fails

• 4 Linux-mainline testmgr serpent KATs (0-bit key, 128, 256, and the …80 key), enc+dec.
• 169 KATs extracted from Botan's verified serpent corpus (58/55/56 for 128/192/256-bit keys, incl. multi-block up to 1 KB), enc+dec.
• 6000 random enc→dec round-trips + 1000x iterated chains per key size.
• The 121/160-bit short-key padding branch was exercised in an earlier revision via direct serpent_set_key calls and cross-checked against the spec padding rule (l_key[kbits/32] = (l_key[…] & mask) | bit); the published corpus covers the lengths cryptoapi actually passes.

twofish.c — 15018 checks, 0 fails (pass-2: clean)

• 6 published KATs (official ecb_ival-derived zero-key/0123…77/0123…ff + 3 LibTomCrypt self-test vectors), enc+dec.
• 1000x iterated encrypt/decrypt chains per vector.
• 15000 random round-trips across 128/192/256-bit keys.
• Known findings DF-0965/DF-2371 (keylen validation — cryptoapi validates before calling, unreachable in-tree) and DF-0966 (me_key/ mo_key locals unzeroed) were NOT re-derived; no new defects found: h_fun/gen_mk_tab table writes bounded for k_len ∈ {2,3,4}; cryptoapi_cipher_freesession wipes the whole ctx (0xFF + explicit_bzero), so l_key/s_key/mk_tab do not leak.

rijndael-api-fst.c — 208 checks, 0 fails (pass-2: clean)

• Reachability: zero in-tree callers (only test00.c, a userland test, references the API; opencrypto/wlan use rijndael-api.c / rijndael-alg-fst.c directly). The file is compiled into the crypto build (sys/conf/files:2270) but dead.
• FIPS-197 ECB KATs (3 key sizes) + RFC 3602 CBC cases #1-#4 through blockEncrypt/blockDecrypt; 192 padEncrypt/padDecrypt round-trips; CFB1 single-block round-trip under the (documented-by-behavior) in-place API contract.
• Known pass-1 findings confirmed live for the record: DF-0968 (blockEncrypt CBC with inputLen=64 bits: returns 0 but wrote a full 16-byte ciphertext block, clobbering a canary placed after the contract-sized 8-byte output buffer) and DF-0969 (padDecrypt ECB rejects the valid padLen==16 that padEncrypt generates — 6/6 exact-multiple inputs).
• The KAME CFB1 branch never reads input in the encrypt direction and requires the plaintext preloaded in outBuffer (AESAVS/MOVS-era API contract) — a single-block round-trip passes once used correctly; the multi-block non-advancing bug remains DF-2369 (known).
• Nothing new memory-unsafe beyond the known set; no info leak beyond DF-0967/DF-2370 (raw key retained in keyMaterial/stack).

chachapoly.c — 22 checks, 0 fails + live kernel e2e (pass-2: clean)

• RFC 8439 §2.8.2: chacha20poly1305_encrypt reproduces the reference ct AND tag byte-exact (proves poly-key derivation at counter 0, IETF nonce split into words 13-15, message stream from counter 1, MAC construction incl. AD/data padding and LE64 lengths).
• RFC 8439 A.5 (265-byte ct + tag): decrypt verifies, re-encrypt reproduces ct‖tag; tampered ct/tag rejected.
• draft-irtf-cfrg-xchacha A.1: enc ct+tag byte-exact, dec round-trip, tamper rejection (hchacha20 subkey derivation + 4-zero-prefixed nonce).
• 264 boundary round-trips (22 lengths 0..4096 × 4 AD lengths × {chacha, xchacha} × {separate, in-place}) — exercises the update(64k)/final(tail) split at every 64-byte boundary.
• mbuf API through DFly-faithful bounds-aborting shims (shim_mbuf.h mirrors m_copydata/m_copyback/m_getptr from sys/kern/uipc_mbuf.c): 9 lengths × 6 fragmentation patterns byte-match the contiguous result; decrypt trims the tag; mid-chain ciphertext corruption → EBADMSG; 8-byte and tag-only packets → EBADMSG. No shim abort fired anywhere in _chacha20poly1305_mbuf.
• Live kernel end-to-end (wg_e2e.log): kldload if_wg, two wg(4) peers over loopback UDP (handshake = xchacha20poly1305 cookie + chacha20poly1305 handshake AEAD; data = chacha20poly1305_encrypt/ decrypt_mbuf on real mbuf chains): 200/200 small + 20/20 × 1300-byte pings, 0% loss.
• Hunted, not found (negative results): nonce reuse at wg rekey — wg_noise.c enforces REKEY_AFTER_MESSAGES (2^60) / REJECT_AFTER_MESSAGES both directions (wg_noise.c:62-63,874,901-902) and the transport nonce is a 64-bit LE counter (wg_noise.c:996-997); cookie nonces are 24 fresh random bytes (wg_cookie.c:231). Tag comparison is timingsafe_bcmp (chachapoly.c:195) — constant-time. The 64→32-bit chacha_encrypt_bytes length truncation (DF-2364) and plaintext-before-tag-check (DF-2365) and poly_key-on-stack (DF-0970/DF-2363) remain the known set; nothing new. Harness debugging during this run initially suggested chacha/poly1305 anomalies (wrong tags) — all were traced to harness bugs (passing ct-length instead of ct+tag-length to decrypt; XORing keystream into non-zeroed in-place buffers; reading a vector from a mis-extracted awk range). Both primitives were additionally cross-verified against a pure-Python RFC 8439 Poly1305 oracle and the RFC's printed A.5 poly-key.

Bottom line

• DF-3562 (new, Low, certain, memcorrupt bucket, unreachable in-tree): one-sided key-length validation in serpent_set_key() → u32 wraparound → runaway/wild OOB writes. Demonstrated live 3/3, fixed and re-verified (0/3, KATs 6349/0).
• twofish.c, rijndael-api-fst.c, chachapoly.c: no new findings; correctness proven against published vectors (15018/208/22 checks, 0 fails) plus a live kernel wg(4) tunnel; two known rijndael-api findings (DF-0968, DF-0969) confirmed live as evidence for the existing records.

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.