Gotta agree with the gist of these responses. This guy is a loser, who disrespects you and is trying to pull you down to his level. You work in the same office? That is always awkward, because everyone else DOES know what's going on.

You need to get rid of this guy now, and you may very well need to find another job. You have a decision to make which will likely affect you for the rest of your life.

God bless you and good luck!