I kind of feel like this too.. I used to feel really bad about it at first, and I've gotten over it, some how.
I think with me it's that I didn't do any of the stuff he did. I was kinda jealous that he got to "date around" and I didn't. I had the chance, but I was too shy or had insecurities or my standards were too high. I was jealous that he had some experience and I hadn't.
I guess it always bothered me that the only thing I ever did was kiss because it made me feel like a prude, though it was the only thing I was comfortable doing.
I was also jealous of the girls.. and that part is hard to get over too, but for me the easiest thing to do is to not think about it.
And like bonaire said, "he probably also ate food, drank liquids and even wore fancy clothes from time to time."
It's funny but true.. you can't be mad at someone for something they did before you.