cc -O2 -pipe -D_KERNEL -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wold-style-definition -Wmissing-include-dirs -Wno-pointer-sign -Winit-self -Wundef -Wno-unused-but-set-variable -std=gnu11 -Werror -DKLD_MODULE -nostdinc -I. -Idragonfly -I/root/dfwb3/forwarder_x86_64/include -Idragonfly/../include -Wold-style-declaration -fno-common -ffreestanding -fno-stack-protector -fno-strict-aliasing -fno-strict-overflow -fno-asynchronous-unwind-tables -fno-omit-frame-pointer -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -mno-fp-ret-in-387 -mindirect-branch=thunk-inline -c dfwb3_rig.c wb_rxeof=0xffffffff808dae30 wb_init=0xffffffff808da6d0 wb_stop=0xffffffff808d9850 wb_start=0xffffffff808d9e90 wb_txeof.part.4=0xffffffff808da490 wb_watchdog=0xffffffff808dadc0 mbstat=0xffffffff8131dec0 mbtypes=0xffffffff81325ec0 kern.dfwb3.rxeof: 0x0000000000000000 -> 0xffffffff808dae30 kern.dfwb3.init: 0x0000000000000000 -> 0xffffffff808da6d0 kern.dfwb3.stop: 0x0000000000000000 -> 0xffffffff808d9850 kern.dfwb3.start: 0x0000000000000000 -> 0xffffffff808d9e90 kern.dfwb3.txeof: 0x0000000000000000 -> 0xffffffff808da490 kern.dfwb3.watchdog: 0x0000000000000000 -> 0xffffffff808dadc0 kern.dfwb3.mbstat: 0x0000000000000000 -> 0xffffffff8131dec0 kern.dfwb3.mbtypes: 0x0000000000000000 -> 0xffffffff81325ec0 === go4270: REAL wb_start under cluster exhaustion (DF-4270) === kern.dfwb3.go4270: 0 -> 0 GO_RC=0 === marker summary === dfwb3: cluster pool exhausted after 34655 clusters dfwb3: saved-chain walk: 17 mbufs intact (m_len==128, marker ok: 0 damaged) dfwb3: after REAL wb_txeof.part.4 with a chip-completed head descriptor: opackets +0, wb_tx_head=0xffffffff82650348 STILL SET, chain[0].wb_mbuf=0 M-bM^@M^T the :1062 walk 'while (head->wb_mbuf != NULL)' never entered: TX completion WEDGED (5s watchdog is the only exit) dfwb3: *** DF-4270 WEDGE CONFIRMED *** dfwb3: go4270 SURVIVED M-bM^@M^T guest stays up dfwb3: cluster pool exhausted after 34355 clusters dfwb3: saved-chain walk: 17 mbufs intact (m_len==128, marker ok: 0 damaged) dfwb3: *** DF-4270 LEAK CONFIRMED: exactly the 17 mbufs of the dequeued packet chain are still allocated and referenced by NOTHING (queue empty, descriptor wb_mbuf NULL) *** dfwb3: after REAL wb_txeof.part.4 with a chip-completed head descriptor: opackets +0, wb_tx_head=0xffffffff82650348 STILL SET, chain[0].wb_mbuf=0 M-bM^@M^T the :1062 walk 'while (head->wb_mbuf != NULL)' never entered: TX completion WEDGED (5s watchdog is the only exit) dfwb3: *** DF-4270 WEDGE CONFIRMED *** dfwb3: go4270 SURVIVED M-bM^@M^T guest stays up ---DMESG--- dfwb3: === DF-4270: force the wb_encap MCLGET-failure path under the REAL wb_start === dfwb3: crafted 17-mbuf chain, pkthdr.len=2176 (>16 fragments -> wb_encap coalesce branch; len > MHLEN -> MCLGET required) dfwb3: baseline accounting: mbtypes[MT_DATA]=344 mbstat.m_drops=0 (chain included in baseline) dfwb3: accounting self-test: +1 alloc observed: 1, back to 344: 1 dfwb3: cluster pool exhausted after 34655 clusters dfwb3: post-drain mbstat.m_drops=1 (+1 M-bM^@M^T MCLGET failure signature, uipc_mbuf.c:1249) mbtypes=34999 (expect baseline+stash=34999) dfwb3: ifsq_classic_enqueue -> 0 (0 = ok); mbtypes now 34999 (enqueue must not change it) dfwb3: dispatching REAL wb_start (wb_encap is inlined into it; return value ignored at if_wb.c:1327)... dfwb3: wb_start mbuf delta: 0 (LEAK expects +17; 0 means the chain was freed INSIDE wb_start!) dfwb3: post-dispatch mbstat.m_drops=2 (+1 since pre-dispatch) dfwb3: post-dispatch probe: MGETHDR ok; MCLGET FAILS dfwb3: post-wb_start state: dfwb3: wb_tx_head=0xffffffff82650348 (== &chain[0]: 1) wb_tx_tail=0xffffffff82650348 wb_tx_free=0xffffffff82650368 dfwb3: chain[0].wb_mbuf=0 (NULL: 1 M-bM^@M^T wb_encap returned BEFORE :1283 set it) dfwb3: TX queue len=0 (packet dequeued and gone) dfwb3: frag[0]: status=0x80000000 ctl=0xa1000080 (FIRSTFRAG:1 TLINK:1 FINT:1 LASTFRAG:0) dfwb3: frags with OWN set: 16/16; frags with LASTFRAG set: 0/16 (unterminated chain: YES) dfwb3: frag[15].wb_next=0x29c53600 vs vtophys(&frag[16])=0x29c53600 (one-past-the-array walk) dfwb3: MMIO WB_TXSTART=0xffffffff (0xFFFFFFFF: chip told to TRANSMIT the half-armed descriptor, if_wb.c:1355) dfwb3: if_timer=5 (5s watchdog armed) dfwb3: accounting after evidence: mbtypes[MT_DATA]=34999 (baseline 344 + 34655 drain stash + LEAK 0) dfwb3: saved-chain walk: 17 mbufs intact (m_len==128, marker ok: 0 damaged) dfwb3: after REAL wb_txeof.part.4 with a chip-completed head descriptor: opackets +0, wb_tx_head=0xffffffff82650348 STILL SET, chain[0].wb_mbuf=0 M-bM^@M^T the :1062 walk 'while (head->wb_mbuf != NULL)' never entered: TX completion WEDGED (5s watchdog is the only exit) dfwb3: *** DF-4270 WEDGE CONFIRMED *** dfwb3: dispatching REAL wb_watchdog (the 5s recovery path)... dfwb3: after watchdog: wb_tx_head=0 (recovered: 1) mbtypes[MT_DATA]=34999 M-bM^@M^T leaked mbufs STILL leaked: no dfwb3: cleanup: chain NOT freed by wb_start per accounting M-bM^@M^T NOT double-freeing; freeing stash only dfwb3: rig cleanup complete: mbtypes[MT_DATA]=280 (baseline 344, delta -64 M-bM^@M^T accounting restored) dfwb3: go4270 SURVIVED M-bM^@M^T guest stays up dfwb3: layout: sizeof(wb_softc)=0x1e80 if_timer=0x6a bhandle=0x3c0 btag=0x3c8 ldata=0x400 rx_chain0=0x448 stat_timer=0x1e68 snd.altq_subq=0x210 dfwb3: resolved: rxeof=0xffffffff808dae30 init=0xffffffff808da6d0 stop=0xffffffff808d9850 start=0xffffffff808d9e90 txeof=0xffffffff808da490 watchdog=0xffffffff808dadc0 mbstat=0xffffffff8131dec0 mbtypes=0xffffffff81325ec0 dfwb3: mbstat stride sanity: sizeof(struct mbstat)=0x80 cachealign-stride=0x80 m_msize0=512 m_msize1=512 dfwb3: === DF-4270: force the wb_encap MCLGET-failure path under the REAL wb_start === dfwb3: crafted 17-mbuf chain, pkthdr.len=2176 (>16 fragments -> wb_encap coalesce branch; len > MHLEN -> MCLGET required) dfwb3: baseline accounting: mbtypes[MT_DATA]=361 mbstat.m_drops=3 (chain included in baseline) dfwb3: accounting self-test: +1 alloc observed: 1, back to 361: 1 dfwb3: cluster pool exhausted after 34355 clusters dfwb3: post-drain mbstat.m_drops=4 (+1 M-bM^@M^T MCLGET failure signature, uipc_mbuf.c:1249) mbtypes=34716 (expect baseline+stash=34716) dfwb3: ifsq_classic_enqueue -> 0 (0 = ok); mbtypes now 34716 (enqueue must not change it) dfwb3: dispatching REAL wb_start (wb_encap is inlined into it; return value ignored at if_wb.c:1327)... dfwb3: wb_start mbuf delta: 0 (LEAK expects +17; 0 means the chain was freed INSIDE wb_start!) dfwb3: post-dispatch mbstat.m_drops=5 (+1 since pre-dispatch) dfwb3: post-dispatch probe: MGETHDR ok; MCLGET FAILS dfwb3: post-wb_start state: dfwb3: wb_tx_head=0xffffffff82650348 (== &chain[0]: 1) wb_tx_tail=0xffffffff82650348 wb_tx_free=0xffffffff82650368 dfwb3: chain[0].wb_mbuf=0 (NULL: 1 M-bM^@M^T wb_encap returned BEFORE :1283 set it) dfwb3: TX queue len=0 (packet dequeued and gone) dfwb3: frag[0]: status=0x80000000 ctl=0xa1000080 (FIRSTFRAG:1 TLINK:1 FINT:1 LASTFRAG:0) dfwb3: frags with OWN set: 16/16; frags with LASTFRAG set: 0/16 (unterminated chain: YES) dfwb3: frag[15].wb_next=0x282a9600 vs vtophys(&frag[16])=0x282a9600 (one-past-the-array walk) dfwb3: MMIO WB_TXSTART=0xffffffff (0xFFFFFFFF: chip told to TRANSMIT the half-armed descriptor, if_wb.c:1355) dfwb3: if_timer=5 (5s watchdog armed) dfwb3: accounting after evidence: mbtypes[MT_DATA]=34716 (chain-inclusive baseline 361 - 17 chain + 34355 stash + LEAK 17) dfwb3: saved-chain walk: 17 mbufs intact (m_len==128, marker ok: 0 damaged) dfwb3: *** DF-4270 LEAK CONFIRMED: exactly the 17 mbufs of the dequeued packet chain are still allocated and referenced by NOTHING (queue empty, descriptor wb_mbuf NULL) *** dfwb3: after REAL wb_txeof.part.4 with a chip-completed head descriptor: opackets +0, wb_tx_head=0xffffffff82650348 STILL SET, chain[0].wb_mbuf=0 M-bM^@M^T the :1062 walk 'while (head->wb_mbuf != NULL)' never entered: TX completion WEDGED (5s watchdog is the only exit) dfwb3: *** DF-4270 WEDGE CONFIRMED *** dfwb3: dispatching REAL wb_watchdog (the 5s recovery path)... dfwb3: after watchdog: wb_tx_head=0 (recovered: 1) mbtypes[MT_DATA]=34716 M-bM^@M^T leaked mbufs STILL leaked: YES dfwb3: rig cleanup complete: mbtypes[MT_DATA]=280 (baseline 361, delta -81 M-bM^@M^T accounting restored) dfwb3: go4270 SURVIVED M-bM^@M^T guest stays up