This sounds like a lack of confidence. I think the best cure is to hit the gym gain some muscle and you'll feel more confident that's what I did when I had this fear after a year I never had it again and people respected me a lot more I guess because I was a lot bigger. So that's what I think you should do.