Some men are very compassionate and understanding. Take my father for example... My mothers moodswings can make her very difficult to be around sometimes, but he is so kind and patient with her, they've been married 27 years. He has always did his best by me too.
Don't worry about bringing him down, he may be much stronger than you think. Do what will make you happy, and if he makes you happy then you should be together