When we feel like this, we might be attracting what we are or the mirror (opposite) of what we are. When you fight or think you want to leave, my advice is to withdraw from him/don't react and instead of focussing on the relationship--focus on loving yourself, finding a project or work that is fulfilling, do things for yourself--fitness, nutrition, meditation, reading, etc. It is an illusion to think that others can make us happy. Focus on yourself and not him because none of us are perfect. Find happiness within and then what you need to do regarding this relationship will become less confusing.