I understand where you are coming from...and yes if you enjoy the relationship I say go for it...

There will always be people that disagree with you and how you choose to live your life and yes they will judge but unless they have walked in your shoes then they have no say in how you live it.