No, it's not selfish whatsoever to think this way. I think it's easy for many people to feel like we SHOULD be doing X,Y or Z. So we do X,Y or Z to feel like we're doing a good deed for ourselves, but in the end, we're doing ourselves no favors because it's not what we really want. So why bother, right?

Do only what you really want to do, not what you feel you should be doing, and you will be just fine.

As for the guy, glad you feel better & right on. A sex starved man who won't push you? GOOD. At least he's not needy about it.