I don't understand why you can't grow where you are if you want? (bloom where you plant yourself :-)

Growing usually means "deepening" as well, but that you don't see your husband as a rescuer or father figure anymore, I don't see that you have yet had a relationship with your husband as he really is? That you are tempted to go with another man (and care about this new man's feelings when you cannot possibly know the new man very well yet) without having yet learned who the man you are with "is"; I don't see any "growth" in just changing men. Rejecting fantasies one has about another person is not the same as learning who one's self or the other person are. If you know who you are there wouldn't be so many questions, I don't think. Were I in your shoes, I would stay and explore myself and my husband/children with them.