I think it might help to try and understand why both of you feel the way you do. You feel lonely, but he doesn't respond well with spending time together. I'd assume in both cases there's something that's lacking to make you both feel this way. In your case, I think you need to understand why you feel lonely. You say he's the only one who puts up with you, which suggests you're having issues with other people in your life?

In his case, it may come down to the stress of work and possibly feeling exhausted and therefore desperately seeking space as often as he can.

The first step in my mind is to try and come to terms and confront your individual issues first, after which you can re-evaluate your marriage and see whether things have improved or whether the relationship needs work to bring back a spark.