DF-5830 / run.3.log
DF-5830 — FULL-WEDGE RUN (guest was reset with-src afterwards; this is why
`guest_dirty: 1` is recorded even though the final state was cleaned).
Launched detached as unprivileged maxx on the stock kernel
(after the 60007-mbuf state above; pool capacity ~146632):
maxx$ nohup /tmp/df5830/leaksendmsg 3000000 > /tmp/df5830/wedge.log 2>&1 &
Polling (each poll = new root ssh exec):
poll1: 154460/146632 mbufs in use (current/max): <- cap passed, frozen
poll2: 154460/146632 mbufs in use (current/max):
poll3: 154460/146632 mbufs in use (current/max):
poll4: 154460/146632 mbufs in use (current/max):
poll5: 154460/146632 mbufs in use (current/max):
poll6: 154460/146632 mbufs in use (current/max):
poll7: 154460/146632 mbufs in use (current/max):
poll8: 154460/146632 mbufs in use (current/max):
poll9: 154460/146632 mbufs in use (current/max):
poll10:154460/146632 mbufs in use (current/max):
(counter frozen: the leaking process is blocked in m_get(M_WAITOK) inside
sys_sendmsg — objcache exhausted, no mbuf can ever be allocated again
because none of the 154453 leaked mbufs will ever be freed)
Serial console (dfbsd-qemu/vm.sh log):
login: Warning: objcache(mbuf) exhausted on cpu0!
Warning: objcache(mbuf) exhausted on cpu5!
Warning: objcache(mbuf) exhausted on cpu4!
Warning: objcache(mbuf) exhausted on cpu1!
Process state and SIGKILL immunity (root ssh exec still possible over the
already-established link, but the system's mbuf pool is dead):
root# ps -o pid,stat,wchan,command -ax | grep leaksendmsg
983 D0 objcache /tmp/df5830/leaksendmsg 3000000
root# kill -9 983
root# ps -o pid,stat,wchan,command -ax | grep leaksendmsg
983 D0 objcache /tmp/df5830/leaksendmsg 3000000
STILL_ALIVE_AFTER_KILL9
Kernel's own attribution of every leaked mbuf:
root# netstat -m
154460/146632 mbufs in use (current/max):
256/33296 mbuf clusters in use (current/max):
0/8260 mbuf jumbo clusters in use (current/max)
263 mbufs and mbuf clusters allocated to data
154453 mbufs and mbuf clusters allocated to ancillary data
77742 Kbytes allocated to network (55% of mb_map in use)
133 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
-> 154453 of 154460 mbufs are "allocated to ancillary data" (MT_CONTROL,
i.e. the leaked sendmsg control mbufs); 77 MB of kernel network memory
is permanently unreclaimable; 133 kernel allocation requests were
DENIED system-wide; the wedged process is immune to SIGKILL (state D,
wchan objcache). Only a reboot recovers.
Wall-clock: the 3x20000-iteration rounds each complete in ~2-3 s; the pool
(~146k mbufs / ~37 MB) is exhausted by a single unprivileged process in well
under a minute of looping.