I don't think this is a matter of selfishness.
From my perspective, it is a matter of communication, or maybe the lack thereof.
Take it from someone with a lot of experience....relationships without communication are destined to fail.
My recommendation is that the two of you sit down and have an open, honest discussion. Don't hold back your feelings.
Talk about your feelings, your fears, your thoughts.
Let him do the same.
If you can communicate, you can work through this.

BOM