I'm mostly frustrated when he doesn't understand something that I just stated before. For example I might say "X doesn't bother me" and then he asks whether X bothers me... But I think frustration is part of any social interaction, and it's fine.