To me it was presumptuous of him to comment on you and also unkind/unwelcoming with you being a guest in his house.

As Rose said, there is often the (challenging) option to let the words slide off of you and not respond, or respond minimally.