You didn't say how old you are. Age can make a huge difference in how women and men feel fulfilled both sexually and emotionally. Is he exhausted at the end of a day, too tired to put more into it so to speak?

Sexual issues in a marriage, as other have posted, are seldom the real root of a marriage ending. There are usually other issues you haven't been able to address or don't see. "It just seems to be the only reason I might not be able to continue the marriage" makes it clear you are looking to get out. Keep in mind there are three reasons to not get out, your children. Have you sought counseling?

Your spouse obviously wants things to be better or he wouldn't have sought medical help for the PE. Keep in mind he may be feeling inadequate and embarrassed, you have left him before and he knows you were with other men. That only adds to the problem. Adding some spice to the bedroom, delicately of course, with some toys, a movie, even getting online in some cases can help. He can only relate to you if you are honest and gently with him in telling him what you like and how you feel. But don't wait for a fight or when you are in the middle of sex to bring up the issue.

Date nights are a good way to give an extended pleasure to your "three minutes". And not to be gross, but his hands aren't broke, .