I am so sorry to hear this. I also have BPD, anger issues in relationships and want to break up with someone with BD. I fear he would be too hurt and I can't initiate anything because of guilt and fear.
I hope this will end well for you and the harsh feelings will calm down. Working on your health is great, and I hope the time will soothe the pain for you and him (he was wrong in this case, fear mustn't be present in healthy relationships).