My own life experience has shown me, that it's possible, in life to have loved more than one person. Each holds their own special place. Yet, doesn't mean, in the bigger picture that they were the right person for the long haul, they carried through certain stages, in life.
There's a word that came to mind, when I read of your friend slotting you into a passive aggressive category...projection. I've also learned that others will categorize instead of owning their own role.
I can read hurt and confusion and angst in your words. It's not easy, to let go of those that bring about such feelings. It's easy to self blame, i.e., your expressed feelings of guilt. I ask, does a truly loving relationship bring forth feelings of shame? Does a truly loving relationship chop down at ones self esteem?
That old cliché, there's plenty of fish, in the sea, does hold truth. I've spent nearly a lifetime making sense of clichés. My advice, hold out hope. Spend time focusing energy on yourself, defining yourself, solidifying your individual identity. An identity that oozes self confidence. You'll surprise yourself for such a feat, eventually.