That is assuming that they would want the same. For all you know, your wife may be having thoughts of leaving you for him. A rough but likely idea. It is a matter of choice between the three of you. I suggest you sit down and think about how such a situation would affect you and your marriage. It could make or break it. It is a very risky situation in my opinion, causing a lot of feelings to change.