I'm glad it doesn't bother you anymore....but are you absolutely sure he knew what "insufferable" meant when he said it? Sometimes people really do just use words incorrectly. I was just watching a reality show the other day, and the husband was describing his 6 year old daughter, and used the word "homely". And his wife was like, Wha?? She's not homely! And it turns out he meant something like sweet or cute, and just used the word incorrectly. It could be something as simple as that.

It's just the other details of your post make me think "unbearable" is not what he meant. For example, him saying he "enjoys" the other 6, don't correspond with him finding them "unbearable." Or him offering you a letter. Or him talking derogatorily about other clients in front of you, which would be totally unethical IMO. Please do update when you talk to him...now I'm curious what he meant!