Like the others have said, if he continues, just be firm and say that you're not OK with what he's doing. This isn't okay, if you had wanted to be doing things with him you would have said yes already.

Be firm that you aren't interested. Consider blocking all communication with him that isn't work related and maybe tell him that it isn't personal.