DF-5436 — ulpt(4) cross-cdev open TOCTOU on sc_fflags breaks exclusive open

What (one line)

ulpt_attach() creates two independent cdev pairs for one softc — /dev/ulptN (ulpt_fifo_methods) and /dev/unlptN (unlpt_fifo_methods) (sys/bus/u4b/serial/ulpt.c:625-638) — and the only thing making open/close of that softc exclusive is the lock-free sc->sc_fflags gate in unlpt_open() (ulpt.c:427) / bookkeeping (ulpt.c:456) / clearing (ulpt.c:465). The USB framework serializes opens per fifo/vnode (usb_fifo_open's curr_cpd check, usb_dev.c:699-710, and the devfs per-vnode lock), not per softc, so two processes opening /dev/ulpt0 and /dev/unlpt0 in the same direction race straight through the gate. The window between check and commit contains usbd_xfer_set_stall() + usb_fifo_alloc_buffer() (a ~64 KB usb_alloc_mbufs kmalloc, ulpt.c:435-437) — measured 27 µs average in the harness.

Consequences when the race is won (all demonstrated):

1. Exclusive open broken: both opens succeed; the loser's session is silently dead (data flows only to sc_fifo_open[dir]'s last writer, ulpt.c:441/454) — its read() blocks forever.
2. sc_fflags desync: whichever session closes first clears the shared bit while the other is still open (ulpt.c:465) — a third open of that direction then succeeds while a session is live.
3. ulpt_reset() fires while the device is open (the sc_fflags == 0 guard, ulpt.c:413, is the desynced bit) — SOFT_RESET class requests issued into an active session.
4. Watchdog polls device status during an open session (ulpt.c:730 guard defeated) — the exact printer-firmware-bug condition the guard exists to avoid (ulpt.c:726-729).

Unprivileged reachability: /dev/ulptN is created 0644 root:operator (ulpt.c:628,635), so O_RDONLY — the direction the racer uses — is available to any local user; O_WRONLY needs operator/root.

Evidence class

No usb-printer device model exists in the guest QEMU 7.2 (checked -device help) and the guest has no physical printer → KLD harness with production code (same class as DF-5383/DF-5367): unlpt_open, ulpt_open, ulpt_close and the ulpt_watchdog condition are copied verbatim from ulpt.c into hx5436_openrace.c, with only the usbd_*/usb_fifo_* calls stubbed at equivalent cost (the buffer alloc performs the production-shaped 2x32 KB kmalloc; no artificial sleeps). Framework reality is modelled: per-cdev serialization only (devfs per-vnode lock + usb_fifo_open curr_cpd) and D_MPSAFE dev_ops exactly like the production usb_ops (usb_dev.c:133-134) — without D_MPSAFE the kernel BGL serializes all opens and the race cannot fire.

Build (in guest)

mkdir -p /root/df5436
cc -O2 -pthread -o /root/df5436/hx5436_race hx5436_race.c
mkdir -p /usr/src/sys/bus/u4b/serial/hx5436
cp hx5436_openrace.c Makefile /usr/src/sys/bus/u4b/serial/hx5436/
cd /usr/src/sys/bus/u4b/serial/hx5436 && make    # -Werror clean
cp hx5436openrace.ko /root/df5436/

(or sh build.sh)

Run (in guest)

sh run.sh

• loads the harness with the production racy gate,
• runs the racer 3x10 s: two CPU-pinned processes free-running open/close on /dev/hx5436a and /dev/hx5436b (O_RDONLY),
• prints the kernel anomaly counters,
• switches hw.hx5436.fixed_gate=1 (the fix.diff mechanism: sc_lock held across gate..commit in open and close) and re-runs 3x10 s.

Expected decisive output

[R] kernel: opens_while_open=53566 ... resets_while_open=103347 ...   (run 1)
[R] kernel: opens_while_open=103639 ... resets_while_open=208422 ...   (run 2)
[R] kernel: opens_while_open=158208 ... resets_while_open=311727 ...   (run 3)
... fixed_gate=1 ...
[R] kernel: opens_while_open=158208 ... resets_while_open=311727 ...   (UNCHANGED x3)

opens_while_open counts successful opens of a direction that was already open — impossible when the gate is correct. Baseline grows by ~50,000 per 10 s run; with the fix mechanism it does not move at all. resets_while_open counts ulpt_open's SOFT_RESET decision taken while a session was actually open. window_crossed (canary re-read of sc_fflags inside the gate window) directly witnesses peer commits inside our own window (3 hits in the 6 s calibration run).

double_open_simul is supersaturated under free-running (both cdevs are almost always mid-open-call) and is NOT used as an oracle.

Note on polls_while_open=0: the 1 Hz watchdog callout has to land inside a microsecond-scale desync window; not observed in these runs — the desync it depends on is proven by the other counters (and by resets_while_open, the same condition evaluated at open-rate instead of 1 Hz).

Artifacts

• hx5436_openrace.c — KLD harness (production code copies)
• hx5436_race.c — userland racer (pinned free-running processes)
• run.log — full baseline-vs-fix run (decisive)
• build.log — full untrimmed module build (-Werror clean)
• fix.diff — production fix against sys/bus/u4b/serial/ulpt.c
• panic-early-harness.txt — boot-log excerpt of the early harness-v1 guest panic (see VERDICT.md for why it is a harness artifact, and what it nonetheless proves)

DF-5436 VERDICT

Status: reproduced (mechanism-level, production code in KLD harness). Impact: dos (broken exclusive open, dead reader session, reset/status poll during open session). No memory corruption demonstrated in the production path.

Why reproduced

The exact production logic at fault — unlpt_open()'s lock-free sc_fflags gate (ulpt.c:427), its delayed commit (ulpt.c:456), the per-fifo sc_fifo_open[] assignment (ulpt.c:441/454), ulpt_close()'s unlocked bit clear (ulpt.c:465), ulpt_open()'s reset guard (ulpt.c:413) and ulpt_watchdog()'s sc_fflags == 0 condition (ulpt.c:730) — was compiled into a KLD with only the usb calls stubbed at equivalent cost. The framework contract was modelled from the production source: per-cdev serialization (devfs per-vnode lock + usb_fifo_open's curr_cpd, usb_dev.c:699-710) and nothing wider; D_MPSAFE dev_ops exactly like production usb_ops (usb_dev.c:133) so the kernel BGL does not serialize the two opens (without D_MPSAFE the race cannot fire at all — verified empirically, see poc_changes).

With two CPU-pinned processes free-running open/close on the two cdevs of one softc (the /dev/ulpt0 vs /dev/unlpt0 pair, O_RDONLY — the unprivileged direction on the 0644 production nodes):

• opens_while_open grew 53566 -> 103639 -> 158208 over three 10 s runs (~50,000 exclusive-open violations per 10 s); every one of those opens returned SUCCESS for a direction that was already open.
• window_crossed = 3 in a 6 s calibration run: direct canary proof of a peer commit landing inside our own gate window (the load measured 27 us average gate->commit).
• resets_while_open grew to 311727: ulpt_open's SOFT_RESET decision (sc_fflags == 0, ulpt.c:413) taken while a session was actually open — the reset guard is defeated at open-rate.
• With hw.hx5436.fixed_gate=1 (the fix.diff mechanism: sc_lock held across gate..commit in open and close, plus sc_fifo_open[] = NULL on close), three further 10 s runs added zero new opens_while_open and zero new resets_while_open; the EBUSY rate rose from ~6k to ~80k per run (the gate correctly rejecting instead of losing).

End-to-end with a physical/QEMU USB printer is not possible on this guest (QEMU 7.2 has no usb-printer device — verified via qemu-system-x86_64 -device help), hence the KLD-harness evidence class; the raced code is the production source text.

Downstream effects (production reasoning, cited)

• Dead reader session: after a double-open, sc_fifo_open[USB_FIFO_RX] names only the last opener's fifo (ulpt.c:441), so the first session's usb_read() (usb_dev.c:1457-1481) sleeps forever — local DoS of whichever process lost the race (e.g. a status-reading daemon).
• ulpt_reset() into an open session: two SOFT_RESET class requests (ulpt.c:177-187) issued while another fd is streaming — the exact condition the sc_fflags == 0 guard exists to prevent.
• Watchdog status poll during an open session (ulpt.c:730-731) — the documented printer-firmware-bug condition (ulpt.c:726-729).
• The stale sc_fifo_open[] debris (production never NULLs it) redirects the bulk callbacks' data destination between sessions; within this USB stack's gating (sticky USB_ERR_CANCELLED at usb_transfer.c:2570, started-flag check at usb_transfer.c:2397-2403, sc_lock discipline in usb_fifo_close at usb_dev.c:783-857) it does not yield a UAF — the same staleness exists in FreeBSD HEAD (checked, identical ulpt_close) and is latent hardening, not a demonstrable corruption.

The early-harness panic (artifact, not a production claim)

Harness v1's close copy freed the buffer via sc_fifo_open[dir] instead of the closing fifo's own slot — an infidelity vs production (usb_fifo_free_buffer(fifo) frees the argument, ulpt.c:468). Under the now-firing race that produced a double free and panic: assertion "*kup < 0" failed in _kfree (see panic-early-harness.txt). This is a harness bug, fixed before the recorded runs; it nonetheless demonstrates (a) the race fires hard enough to corrupt shared driver state within seconds, and (b) the class of cascade a driver suffers once the gate loses — in any driver that, like the harness, frees through the shared pointer. Production ulpt's wrong-free surface is instead the fifo redirection above.

Fix

fix.diff (validated as hw.hx5436.fixed_gate=1 in the recorded runs): hold sc_lock across the gate..commit section of unlpt_open() and the bookkeeping of ulpt_close(), and clear sc_fifo_open[dir] on close. With the mechanism enabled, zero anomalies across three 10 s contention runs (vs ~50k/10 s baseline).

Not verified / limits

• No physical evil-printer end-to-end run (no device model on guest).
• polls_while_open stayed 0 (1 Hz callout vs microsecond desync windows); the underlying desync is proven by the other counters.
• No memory corruption on the production path (see above for why the stale-pointer trail dead-ends in this stack).

Fix status (batch validation)

Applied with the 36 sibling fix.diffs of this batch in findings/poc/_batch41/all_fixes.patch; single in-guest make -j6 nativekernel KERNCONF=X86_64_GENERIC completed RC=0 with zero errors under -Werror (full log: findings/poc/_batch41/fix_build8.log). Behavior A/B intentionally skipped for this Info-tier item.