It Was rude.
If you are sure it was about you I am going to suggest something that has worked for me in the past, and that is to just get the guy alone and say, without getting emotional "I heard you say"..." about me, if you have a problem with my work, or with me, tell me."---if you can stay calm it generally works. Ignore his poor attitude. Pretend you are speaking to someone you expect to be professional.
Also, I find that even very difficult people, including supervisors (it was usually supervisors for me) are glad you said something to them instead of to their boss first. But, if this is continuous, and a hostile work environment is created then decide whether you can stay as things are, need to look for something new, or go ahead and talk to the manager, and if the manager is not helpful, go to his/her boss.