To avoid leading him on, I'd suggest not inviting him to shared social activities, especially since his interest is mainly with you and not your husband, and your husband doesn't seem to want to hang out with him. If they were buddies, it would be okay if you were careful.