It sounds like you and he have different sex drives. IMO, he's being a jerk. You always have the right to refuse sex. For him to threaten to break up with you over it is abusive and manipulative. Coercive. You are better off without that in your life.