I would say it is natural to think this way. Is it wrong? I think you need to get some input from your friend in order to determine that.

When it comes to relationships, we tend to assume a lot of things and it's easy to accept that as truth. However, a good relationship, whether it be friendship or romantic, requires clear communication. If you are having doubts about what happened, ask him. Relationships require trust. If he isn't understanding or becomes offended, then maybe he isn't the friend you thought he was. You don't have to accuse him of anything, just let him know how you feel about the situation and get some honest feedback. If this is part of a trend you have been seeing with this friend, then it is definitely time to confront it.

From what you wrote, your friend seems polite and did invite you hang out for a couple days. Sure, there is a lot more he could've done to reassure you but that is often a fault we all have. It looks like you just need to talk things out and let him know what your expectations are. None of this has to be done in a mean or hurtful way. You're friends for a reason, let that guide you. Trust in your friendship and build on it. If this kind of a behavior becomes a habit, then yes, it might be time to re-examine if this friendship is worth it. But deal with that when it becomes a definite problem and talking isn't working anymore. For now, hold onto all the friends you can because they aren't always easy to come by. Good luck to you!