Perhaps talking about these feelings with either your wife or your friend would be beneficial. That way, you and the other involved can talk out all of your complex emotions and you no longer have to feel guilty-if you do-over experiencing them.
This might be the sort of thing people get upset about, so I suppose if you think your wife would be upset by this, it would be best to disregard this advice.