Since you've explained your reasoning, I can see how you would feel that way. Hopefully you can resolve the issues you have and not lose this relationship. To me though, there is evidence that some people have ongoing sexual issues and still manage to be reasonably happy in life.