No, you are not a bad person because of this - not at all.

@Bill3 makes an excellent point. I wonder the same thing? What happens after a while is that people could stop inviting you if you continue to say yes, but then decline at the last minute. Would it hurt to join these gatherings and take a risk? We all risk not being liked when socializing. If you're worried about not being liked, just ask people questions about themselves - people love to talk about themselves and their lives, so if you show an interest in them outside of work, it will be well received.