I'm on the same kind of dilemma.
Man-woman relationships are complicated, but in my eyes, that's the beauty of it.
On one hand you're comfortable together. On the other hand, there are details (such as age, past, physical health) which bothers you.
I can recommend defining your relationship together. That way it will be clear for the both of you what you all want in the relationship.
I love the idea of defining the relationship together, setting together what's best for both partners.