Have you discussed this with him? communication is key. sometimes I think we have to act with our mind and not on emotion. You might feel like you are distancing which might keep you from him. But go with your mind and do what it tells you is best.