When you take the bus to see him, you are allowing him to avoid facing his anxiety.

What he needs to do to get better is face the fears. He needs to do what he fears, not withdraw from it. A therapist can help with that. But if he isn't willing to face fears (e.g. by meeting your family) or ask for help, I'm not sure what can be done.