It could have been a very poor choice of words where he had no intention of stirring up such feelings. Maybe he was totally unaware of how you perceived his comment. I would call him when you feel ready, and tell him what bothered you so much in the session. Give him a chance to explain himself and make things right. I'm sure he had no intention to "throw you for a loop".