I think it's very important to be with someone who can accept your MI. While it sounds like he's quite supportive in many respects, behavior such as that you describe might really affect your self-esteem. It's your call, though.