In a word yes. It sounds like he might have been chancing it with you but your husband's reaction has probably scared him off.

If it's unsettling for you maybe block his number on your phone just in case?