I have had to let some friendships go when the friend was judgmental or hurtful in other ways. I have learned that I have a hard time, too, coming up with an appropriate explanation for my leaving that is truthful, yet doesn't blame the other.
I struggle with this too.