I know that size doesn't really matter (except really fat) too much to me when I am gonna do some spontaneous fun but if its planned then they have to be what I want, which is not fat.

To be honest tho do you really care what he thinks? If it's just for some casual fun go in, get yours and get out. Plus how do you know if he isn't having his own insecurities? Don't think too much about stuff you can't change and just go have fun