I think it all depends on you and your partner's routine.. Job nature, work pressure, growing kids and so on ? It will be a litte bit harder if one party has higher sexual urge while the other is not... but it is common if other affairs overpower the intimacy part...

I feel you should talk openly to him how much sex means to you and see how he response. The strange part to me is it doesnt bother him at all when he found out that you cheated on him... In this case, what Rhasody has mentioned would have higher possibility to be true, but no guarantee..

And i believe a couple's affection is not only based on sexuality but many other things.. Such as showing your concern, quality time spending for each other, a simple movie together, a call to say i love you and so on....