I think considering your recent troubles in your marriage and both of your resolution to solve them, he really does need to stop and step back.
His relationship with her doesn't seem very appropriate for a married man. He also acknowledged he would not be comfortable if it were reversed.
He had also admitted to having loving and deep feelings for her. And I wonder if all his intentions are quite all about helping her and not some about his own wants or a bit selfish. Knowing it could hatm her as well as his marriage does seem a bit selfish. Like he is also getting something out of this for himself.
Sometimes the best way to help someone is to direct them to the proper help and remove yourself if it is not really healthy. It sounds tho that he is using her problems as a an excuse to continue doing something he knows is hurtful to your relationship. Justifying it.
It does not sound like your husband is really considering your feelings here. He is putting you in a bad position, and you should not have to tip toe around it in order to feel any security. That is not healthy either.
I'm sorry, that is just how I see it. I think you are being more than fair and giving, maybe too much so. Your needs matter too. I think your feelings on the matter are quite rational and on the impartial part... this is your marriage, you are not supposed to be impartial always.
I am not suggesting your husband is physically cheating but it does sound as tho he may be emotionally having an afair. Or that he is just really stepping over you without reguard to your feelings. He does need to decide if he wants this marriage to work. It is not all up to you.
As far as the friends, they are not the only ones putting him in this position, he does have a choice and a role in that matter. No one can force you to feel or be responsible for another. Not only that it is impossible to be responsible for another adults life.
Your husband might be a great guy, but I don't think he is being a great guy at the moment to anyone involved .It would seem like a more appropriate place to be spending his time would be with his two small children and maybe in couples counselling. That might be a good idea if you two do not already do that.
Has he adressed his drinking and pot use? Has he been any more supportive towards you and the changes you have been making? Has he been more helpful with the children or the other duties ?