Well. The fact remains that he specifically asked about coming over to your house while you were there alone. I don't think that you can necessarily believe what he's saying about your misunderstanding. Since he knows your mother, it makes perfect sense that he would say that when you told him to leave you alone.

I think you responded in exactly the correct way to a married man who's close to twice your age when he asked to come over when your mom wasn't there.