no, it doesn't sound selfish.
It sounds like you are aware of yourself.
Maybe take a bit more time, sometimes, to consider whether you actually want to do what they are asking of you. Then if you still want to do what is asked, that's up to you. But if you don't want to, then figure out how to say that you don't want to.
Friends come and go. Some might keep in touch for many years. Many will move away and you might never hear from them again.
Live to please yourself (and get along with people).