Just be honest with him. You are not fulfilled with the physical side of the relationship, your needs are not met, and he does not want to do anything about his 'problem'.

I would just be straightforward in telling him you don't want to be physical when you are not fulfilled in this way.