I don't think you're a bad person.. I think it's normal to feel this way sometimes, if you're depressed and had a difficult past

Does your friend know that you are depressed? If this thing bothers you so much, maybe you can talk with him about this, explain why it bothers you, and see how it goes from there

Does this friend support you in some way?