S4 is right. He is doing it to watch your reaction and you're not disappointing him. I'm not saying it's easy, but once you're able to walk away and not feed in to what he's trying to do, the better it will become. You mentioned it yourself the other day that when you were able to walk away things were better afterwards. You won't be able to walk away and stay calm every time, but the more often you can do it, the more you won't feed in to what he wants and the easier it will become to walk away as time goes on. Don't give up. Don't get mad at yourself when you just have to let to let go of your anger and irritation, but continue to work on it and things will gradually get better, although probably not nearly as fast as you'd like.