This is only my opinion, but I think you and your friend need to put the notion of being together romantically behind you or break off your friendship. Your husband is understandably worried because he's not the best friend and he could end up losing the lover side as well where we're told spouses are supposed to be our best friends and lovers. He's seeing himself losing both.