Well, I would discuss this with him and your primary mental healthcare worker.

I want to throw this out there.... On what is your friendship based? Evaluate this and consider how important it is having him in your life. If the answer is that his importance is relatively high to you, then cultivating it would be the thing to do; and, that means being forthright and honest. Put the ball in his court then and tell him about the impact of those occasions you get together. If he cringes and walks away, it was time to cut him loose anyway.