I would say heed your gut and really discuss your feelings with him. It sounds as if he has spent a great deal of time with this woman. Also, he seems to have relied on other women emotionally for a long time. Most relationships are what they are based on the emotional connections between the two people. If he's experiencing that with other women, that can definitely make it easier for one of them to weasel their way in. You need to have it out with him and let him know that this is damaging your faith in the relationship.