It's very hard when someone we love doesn't want what we want but no one has to give a "reason" to another as to why they do/do not want something. Were I you, I would back off the pressure some (of needing a reason or being gung ho for "working" on problems, etc.) and see if a relaxed, wooing approach might bring him around?

Think of his words kind of like someone who mentions some idea or wish but doesn't do anything to work towards it. As long as he's not seeing lawyers, making plans, splitting up what's his and what's yours, etc., see if you can be a bit more relaxed about the whole thing and work toward whatever he has expressed as important to him that has caused the arguments/issues?

If he is accusing you of being with or wanting to be with other men, set up nice dinners with foods he likes and just "be" with him (so you obviously can't be wanting to be somewhere else?); being relaxed and "going with the flow" instead of trying to resist his "side", his wrong thoughts or ideas might make enough slack so he'll "fall in" and be willing to re-examine the whole thing whereas your pushing/pulling for "fixing" could just be making him resist harder or feel overwhelmed and tired so he "gives up".