The problem is his, not yours. It is exactly the same as if he had asked you if you have ever driven a car and then proceeded to tell you car horror stories. It's an ignorant and self-serving answer.

Don't carry his behavior with you. Let it roll off of you. Chalk it up to a lack of empathy and move on.

If you're going to be uncomfortable around him, you have two choices. You can let it go or you can talk to him about it. If you talk to him and he's a decent human being, he will apologize for his unthinking actions and you can both move on. If he gets defensive or belittles your concern, you will have to choose whether you can continue to be around him or if you will be able to understand that he is a small person and deal with him on that basis.