My immediate thoughts are to work on your relationship with your husband before considering building secondary relationships. But I don't even begin to understand such a thing. It doesn't cause me feelings of considering it wrong though, I just don't get it. But I don't gett a lot of things.

The two of you got into this relationship with an understanding. That has changed for one party. To me that is a breakdown of the relationship. If he is unwilling to respect your original agreement he is not fullfilling his bargain. Obviously you have two choices; stay and be forced to live what may be a lie, or move on. You don't need permission from anyone to do the latter.

Outline your expectations to him and ask that he do the same for you. Is there an overlap at all? This may provide the answer to your dilemna.

By the way, I am wondering about the agreement you and he had in the first place. Did you speak about conditions?