You stated in your original post that "people talk...". Who are these people & why are you listening to them? Yes people talk, but do you know how many of them lie! Stop listening to "other people" & listen to your core values & life pillars. Who you are.
Do you know how many "people" lose their virginity & regret it? Many! I did.
This 3% that you mention. Is this based on hard scientific facts, a long term study? Again how many people lied & said they had sex...when they didn't.
I think you need to change how you're viewing this thru societies eyes & their pressures. They are not your own.
You're 25yo. I think if you'd double your age & posted this then we'd have an issue to look at.

Sex does not make you a man.