You are not being selfish at all. You’re thinking about your family as a whole. It is difficult to believe that he is accusing you of being selfish when clearly he is the one being selfish.
The only way any man should be touching their wife is with love and affection.
I will keep you in my thoughts while you decide how you will deal with this situation.