No, I don't thing you are selfish. I think we all want to hear those 3 little words at some point with someone we have come to love. Sounds like they are something that bring some anxiety or pain to him. It's something he will have to work out. Unfortunately, that leaves you with a problem. You will either have to let go or accept him the way he is.