I have a similar experience. Most of my abandonment issues aren't around men or relationships. It's not that I want to be alone, but I'm terrified of close, intimate, physical relationships. At the same time, I want it more than anything. I haven't quite figured it out. I tend to chase after people who don't seem to want to like or love me as much as I care for them, and it causes me a great deal of pain.