I wouldn't leave him just yet. I'd take a look into why you are feeling this way. Have you tried to be honest and direct with him? Tell him how you've been feeling???