You still have the abandonment issues if the person sticks around and treats you like crud, is the thing...

But it's harder to accept that someone lived with you or was in your life, and thought it was okay to treat you badly, than to have them not even know you. If they don't know you, you don't end up asking yourself what was wrong with you, why you weren't looking after, how they could live with you and treat you like that.