I am in a somewhat similar position to you in terms of sexual partners.

My first girlfriend (best friend) and I were together for a long time but we never had sex because she wanted us to save ourselves for marriage and she passed away in an accident when she was 17 and I went through depression for a number of years and I ultimately blamed myself even though I wasn’t with her at the time because she was on her way to see me when it happened. After that I didn’t get into another relationship until I met my Wife and I was still a virgin then and I didn’t lose my virginity to her until a number of years into our relationship due to her being raped. Some of my old friends wanted me to have an open relationship but I couldn’t because I loved her and I was willing to give up having sex and remain a virgin forever because I didn’t want to be with anybody else.

Fast forward to today and we are still together and my Wife is pregnant with our second child (which will be her third) and even though she is my first and only sexual partner I no regrets because I never wanted to sleep around with multiple partners and only wanted to have the one woman I loved like I had with my first girlfriend and I have that and my Wife is the most incredible, caring and supportive woman I have ever known. I don’t have the urges you have because I am Demisexual and I need to have a strong emotional connection with a woman to be sexually attracted to them but I think you need to ask yourself how much you really love your Wife. The fact you are becoming drawn to other women may very well be a sign you are falling out of love and growing apart or perhaps you are just being greedy like another member suggested.