I think tolerating him is probably the best you can do, it doesn't sound like there's a lot to like from what you write.

It actually sounds to me a pretty natural response to someone who was mean to your bf when he was little and vulnerable.