You're letting this guy win. You need to stand up for him, tell him that if he doesn't treat you better then you're going to stop talking to him (sounds childish but you know what I mean). Don't let him win. I know what you mean about feeling obliged to be friendly but he sounds awful! You'd be much better without him!
Amy