I do believe in emotional infidelity. But I think it has more to do with developing romantic feelings with someone else and starting a romantic relationship with that person - even without the sex involved. I think you two grew close as family does, instead. Your husband is controlling and probably doesn't want anyone else in your life, so this sounds more like his problem than yours. It sounds like he'd be unhappy if you had a female close friend too.

On the other hand, I have to ask what there is to salvage with your husband. If he's never emotionally available, does not seem to want to be, and is super controlling then why take him back? Is there anything you guys have to work with? He might be acting angry and controlling just because he's getting more and more hurt. I don't think the solution is to never see his brother but to talk it out and see if you can spend more positive time with your husband. If he's not interested in doing that at the very least, then I'm not sure what there is to salvage.