Have you tried talking to him about this to explore how he feels about sex? Four times in a year seems very low to me. But I think that sex drive can vary widely between individuals. Perhaps you could try to weight up the good and bad aspects of this relationship, in terms of things that are important to you. That might help you to be sure whether ending the relationship is the best decision or not.