I also notice what he wears, and sometimes I wonder about it. He dresses much nicer some days than others, and I wonder why. He wore sweaters a lot in the summer, and I wondered about that too. One time he wore a jacket I really liked.

But it doesn't really affect me much. I don't feel more or less connected to him
depending on what he wears.

I wonder why your therapist's clothes affect you in that way. Perhaps certain shirts stir up unconscious memories?