Hi, Bree...
This doesn't sound like a good arrangement to me at all. Like you, I have never expected to be with someone 24/7, and in fact I need my own space. But your husband sounds like he is being selfish and ignoring your relationship and your needs. Telling you "if you had kids, it would be different," sounds cruel also! That should have nothing to do with spending happy time together on weekends and evenings during the week. Your relationship should be his first priority!
Patty