I do not want to say you are "way over-reacting" as that would not be anyone else's call, but it sounds to me like the therapist is just saying he wants to give you an opportunity to discuss your feelings with him, not that he wants you to find a different therapist or anything like that. I think the short message from him is more of an indication he does not want to discuss it in email form. Is he usually more informal in emails?