Good for you! I think you did the right thing for you in speaking up for yourself, even if he doesn't like it and even decides he doesn't want to be around you!
Question: Have you ever had a chance to talk to a counselor? I was one of those people who seemed to look for people to try to fix, too. However, to make a l-o-n-g story short, I have talked to counselors over the years, been able to get better self-esteem and deal with some other issues. Of course, no people are perfect (and if asked, I could tell my current friends things they could work on

), but it is really a relief not to feel like I have to be responsible for the well-being of other folks, to my detriment.
Are you feeling okay about things?