Talk to him about it. He should support you, as you're obviously struggling. Not yell at you, even though he's annoyed. You said you told him, but maybe you should have a serious and long conversation about it.