Are you seeing a therapist? I think it might help to figure out what's behind this desire to be with such man. Do you in general have low standards when it comes to men? I would worry what kind of message you send to your kids, are they young or grown? If they are young girls they will grow up repeating unhealthy patterns (unless you intend to keep him s secret). I'd definitely talk to a therapist