Well, he did ask if u were comfortable with his suggestion.
So I would advise you reply with a polite "no thanks, it would be highly inappropriate for a married man to visit me while I'm alone at home" and leave it at that.
We are being taught boundaries in DBT right now, and apparently there are varying degrees of "no"...
Who knew right?
So you can start off with a passive one and if that doesn't do it, you make them firmer, or even ruder as suggested above.
In relationships / friendships, its important to realize that you have a right to your boundaries and more so, that its up to you to enforce them.
If he wont respect them, no matter how you say no, well then I'd call in the big guns and get mom involved and also block him for good measure.