Agreed. Maybe he was mostly focusing on the "worry" part of your question? Like he read it as "Are you worried about me?" rather than "Do you think about me?" or "Do you care about me?" He could care about you very much without necessarily thinking of you that often. (Or without necessarily telling you that he thinks about you.) I would definitely talk to him about what he said and how it made you feel. I've had some misunderstandings with both my T and marriage counselor recently (and in the past), a few times feeling really hurt by something they said, and talking with them really helped. Even though I had been tempted to run away.

And I think the fact that he's willing to make it work financially for you to see him twice a week shows that he cares...that it's not just about the job or the money.