If you're not the kind of person who wants to have a blended family in the future and be a step-mom, then I'd say this isn't the right relationship for you. Honestly, I have no desire to have step children as I know the drama that it brings with the ex and so on and so forth (been through that before, it wasn't fun, and I'm not doing it again).

Its not stupid to leave someone because of their past. Would you stay with a child molester and give the excuse "well that was in the past!". Nope, you wouldn't. These feelings of yours are very valid and I don't think you can just throw the past away, no matter how minor it may seem, as what matters is how it affects us.