If they are sharing personal issues, and he stated that he respects her more than you, I feel you are justified in how you feel. If you have stated to him that you dont feel comfortable with him sharing personal issues (i'm assuming its regarding the two of you) and he continues to do so, I would be very hurt. Coming from a male perspective, he should of at least tried to reassure you before getting defensive.

It would appear that he has a hard time seeing this from your POV and that is very sad. This is the way that a lot of marriages end up with one partner having an affair, be it emotional or physical. Have you thought about seeking help from a therapist? even if you went by yourself it may help you. Do you have self-esteem issues that may be playing into this? If your gut is telling you something is up, I would trust it.