min() truncates 64-bit size_t to 32-bit u_int then stores in signed int: negative loop count unbounded buffer overrun
Summary
iconv_xlat_conv declares int n r (iconv_xlat.c:86) computes r=n=min(*inbytesleft *outbytesleft) (iconv_xlat.c:95). Kernel min is static __inline u_int min(u_int a u_int b) (sys/sys/libkern.h:76) so 64-bit size_t operands silently truncated to 32-bit u_int then stored in signed int. If truncated value has bit 31 set r becomes negative and while(r--) (iconv_xlat.c:98) runs ~2^32 iterations reading and writing far past both source and destination buffers. Sibling iconv_xlat16_conv avoids this by using size_t ir or and while(ir>0&&or>0). iconv_convmem(handle dst src int size) does inlen=outlen=size with no sign check so negative size becomes huge size_t whose low 32 bits exactly negative int bit pattern triggering runaway. Current smbfs/msdosfs callers pass small positive lengths no live trigger today. If any VFS/future caller passes length derived from attacker data without clamping becomes kernel memory-corruption primitive.
No comments yet.