I agree with many other responses here.... I wouldn't place the blame on you, but on the guys involved even more so. If they break up with you just over you not having sex with them that's wrong. It's your body you have the right to say no.
Granted I have no experience in having a relationship, but it seems like most people are searching for a good relationship... and that that kind of relationship is really hard to find.

That does not mean you're a bad person.

I feel like I hurt people a lot too. I guess all you can do really is try your best to help people the best you can. Your intentions matter.