I understand what you feel....I had same kinda experience before....I got intimate with an ex-boyfriend whom I still love to this day....then I didn't say anything about the relationships that I had between my divorce and meeting him....leaving him thinking that he's the second one in my life and well...he was so proud....then I found out I can't live with this lie....so, all of a sudden, I told him in the bed...."I've been with two other guys besides you!"....he got shocked, but he said..."ok"....that's it....he said why were you afraid of telling me and you lie to me? that was his questions....but later on he dropped it and he wasn't upset....he even was happy that I told him....
I believe this is more for you than him....This is in your mind....He doesn't need to know about couple of not serious relationships that you have....I think you are creating monster in your head....so, if you can't keep the monster in you, then let it out and see what the monster will do? Are you ready to do such a thing?
M