If he has bipolar, this sounds like he is becoming depressed.

Maybe you could say to him 'I'm concerned about your unwillingness to come and hang out. I'm worried that you're becoming depressed. I'm here if you want someone to listen ok?'

EDIT: if he is becoming depressed, you may have to do more than what you might think is your share of the work to keep the friendship going. Don't take on too much, and try and know when to leave alone for a little bit. He will quite easily just not contact you, so you might end up having to go out of your way sometimes.