This is what I would do.

As your wife I would be taking steps to protect myself from his attention ie never accept any gifts from him and never email or text him.

As for him its time you AND your wife had a word with him telling him to back off as you dont appreciate the attention he is giving her. That you have let let alot of things ride in the past however you are now calling a holt to things as the text message was too much for anyone to 'politely' ignore.

He may defend himself with it was just a joke etc, just point out to him how you BOTH didnt take it as a joke and insist it stops right now.