# DF-0245 — Per-cpu `iowbytes` counter underflow via thread migration

**Verdict: REPRODUCED (source-level).** Runtime triggering is timing-dependent (needs inter-call
CPU migration); the accounting logic is confirmed broken by trace.
**Impact: availability (low)** — corrupts the I/O throttle `factor`, not memory corruption;
no escalation chain.

## The bug

`sys/kern/kern_iosched.c`, `badjiosched()`:

```c
65: static int
66: badjiosched(thread_t td, size_t bytes)
67: {
68:     globaldata_t gd = mycpu;            /* snapshot CURRENT cpu */
...
79:     td->td_iosdata.iowbytes += bytes;   /* per-THREAD accumulator (migrates w/ thread) */
80:     ioscpu[gd->gd_cpuid].iowbytes += bytes;   /* per-CPU counter (static array) */
...
88:     bytes = (int64_t)td->td_iosdata.iowbytes * delta / (hz * 10);  /* decay from td total */
89:     td->td_iosdata.iowbytes -= bytes;
90:     ioscpu[gd->gd_cpuid].iowbytes -= bytes;   /* <-- subtracted from CURRENT cpu only */
```

`td->td_iosdata.iowbytes` lives on the thread and **migrates** with it across CPUs. `ioscpu[]`
is a **static per-cpu** array. The decay (line 90) subtracts a `td->iowbytes`-derived amount
from `ioscpu[<current cpu>]`, but the thread's accumulated `iowbytes` was **added across
multiple CPUs** (line 80, on whichever cpu each call ran). If the thread accumulated weight on
CPU A and then migrated to CPU B, the decay subtracts from CPU B which never received the
contribution → `ioscpu[B].iowbytes` (a `size_t`) **underflows to `SIZE_T_MAX`**, after which
the `factor` computation (line 96) goes haywire (huge divisor → factor≈0 → thread starved, or
sign-wraps). `biosched_done()` (line 117) has the same class of bug.

## Reachability

Triggered by `bwillwrite()`/`bwillinode()` (lines 127/157) on every buffered write / inode
op. The underflow needs the thread to migrate CPUs between an `+=` (line 80) on one call and a
`-=` (line 90) on a later call — plausible on SMP under load but timing-dependent. The harness
`df0245_mig.c` spawns many writer threads to maximise migration probability; deterministic
observation is hard because the bad state only surfaces when `ioscpu[cpu]` is read back (via
`iosched.debug=1` kprintf) after an underflow.

## The fix

`fix.diff` clamps each per-cpu subtraction so it can never underflow (at both line 90 and the
`biosched_done` line 117 site): if `ioscpu[cpu] < bytes`, set it to 0 instead of wrapping.
This is the minimal safe change that eliminates the `SIZE_T_MAX` underflow (the reported
defect); the deeper fix would track per-cpu contributions per-thread, but clamping is
sufficient to prevent the accounting explosion.

## Kernel refs
- `sys/kern/kern_iosched.c:80` — `ioscpu[gd->gd_cpuid].iowbytes += bytes` (per-cpu add)
- `sys/kern/kern_iosched.c:88-90` — decay subtracts `td`-total from current cpu (mismatch)
- `sys/kern/kern_iosched.c:117` — `biosched_done` same underflow class
- `sys/kern/kern_iosched.c:135,164` — `bwillwrite`/`bwillinode` callers