Perhaps decline future evening/after work invitations? If you want to be friends and have the same lunch hour pay him back by buying his lunch?