Hi Ticli. Love your avatar.
Do you think the memories you mentioned could still be at play? I know you specified that it happens no matter what is on your mind, but the memories are still there, you know? Have you spoken to a therapist about these things?
You have to understand that you, as an adult, have no reason whatsoever to feel upset or ashamed or whathaveyou for being with someone. You are an adult making a conscious decision to be with someone. You're not doing anything at all that is wrong.

I don't doubt you know that, but I felt like it may be good to hear.
Please take care, and know I'm praying for you.