I think it's because of your childhood that you like to keep your distance from people not to get hurt again, which explains your lack of kindness, as you say. There is no real 'proper relationship'. Just as long as there is a connection, a love. That you still have feelings towards each other.

Look, if you feel uncomfortable, just ask for some space. It's your mind and body - you can allow every affection you like. I think it's only right to be angry or mad at something if the reason behind it isn't valid.

Stay strong anyhow. I'm proud you crawled out of that hole and stayed strong. Good for you!