This guy doesn't sound like much of a friend to me. It sounds like he only wants you around when it's convenient for him. Personally, he sounds like quite a jerk to me, but that's just me. I know how much he means to you.

You might want to just say to him what's on your mind, but don't go into detail. Like ask him if he is your friend or maybe if he wants to be better friends...I don't really know.