I would agree. If sex is important to you, if need to feel satsfied and fullfilled. Even if it isn't about sex, it doesn't sound like this guy even wants to hear your concerns. You have to do what it best for you. It sounds like he doesn't care much about your releationship.