You don't love the guy so why would it be so hard to just tell
him that it's not working for you. He knows he has a physical problem that can probably be medically corrected. This is not YOUR problem. Maybe you need to give this guy a detailed description of what you need to be satisfied.
I have to self pleasure while my SO is laying right next to me.
Can't have intercourse but she does have a mouth and fingers.
Won't help out at all because SHE can't climax. It's a complicated issue I've discussed elsewhere but the bottom line is that for me , like you , it's a dealbreaker. The fact is this is all about clear cut , honest , communication.