Sounds like he knows he does this but is unaware as to what causes him to do so. I wouldn't take it personally at this point since he does it all around and I wouldn't think that it's particularly about you but his aversion to close socializing. It seems to me that he's probably got issues with it and it's his reaction to someone getting too close or him feeling smothered or something similar. Finding out the trigers are up to him but my suggestion is to take a break when he is getting on your nerves, might do you good and also might make him settle down a bit too.