I don't like confrontation either. Why don't you just say that it's been bothering you a little bit that he said <insert what he said>. Don't make a big deal out of it. Just say that you know it's a little thing, but it's been on your mind, so you want to ask him what he meant so you can stop worrying about it. I think that's a fairly mature way to handle it, and I can't imagine any reason in the world why he would not respond to you with an explanation or clarification. That's what therapy is about - to talk about things, and clear up misunderstandings or confusion.

I hope you can find some way to bring this up since it's so obviously on your mind and bothering you.