Not going is not going to get you any help any faster. Maybe you could express to him the things you don't like next time - about "having your number", being uncomfortable with what you're talking about lately, etc etc. I'm sure he's not going to be upset with you about the homework deal.

Next time really try to make it. I hope things go better for you!