Sex is very important and a perfectly good reason to end a marriage - a part of you isn't being cared for and just because it relates to sex that doesn't make it somehow less important. It also sounds like he's witholding affection - i mean he won't even lie down with you and just caress each other? I would have a serious talk with him - tell him that you don't appreciate being demonized for having a healthy libido (i'm referring to the incident whereby he said yours was too high) and that you're considering moving on as you don't appreciate him not taking the issue seriously. If anything it will put the wind up him. Even if it just gets him to be honest once and for all - there's no problem in not wanting as much sex as your partner but if you're not honest about it and you treat the situation like something that doesn't require a lot of attention and care...that's pretty unfair.

I'm by no means suggesting separation but it sounds like this issue is coming to a head anyway, i hope it paves a path for a more positive solution. You could well work something out but it's going to take as much effort from him as it will you. That's what a marriage is. I really feel for you...i hope things work out for the best.