Yeah. Here's the thing. I think this is raising some tiny hair on the back of your neck or you wouldn't have posted this question.

Why is he rushing to make you feel more comfortable? If you are having uncomfortable feelings about your marriage or your attraction to other people, you really need to feel those feelings. He is doing you a great disservice by explaining away your discomfort.

Your prettiness or lack thereof is completely irrelevant here. Of course you're lovely, but if you were older or less pretty, would your attraction to people other than your spouse be somehow less human or less understandable?

It sounds as though you like him and your basic feeling about him is a good one. Perhaps he's a good therapist who had an off moment, they all do occasionally. But let's be clear, it was an off moment. Time will tell if he's a good therapist. But that comment is in no way an example of great therapy. In my opinion.