Men turn down sex more than women do, and for all sorts of reasons. It sounds to me like maybe you haven't resolved your current series of arguments.

How far have you been able to get with him?