No, you shouldn't let it bother you. He seems to be a misanthrope, and since you're human, you just fall into his category of dislike for all things two-legged and breathing.

Just make sure that you don't let him disrespect you.