Do you think skipping sessions like that affects him much? I would think that going there and very directly telling him what bothers you might affect him much more, and more productively, given that he would have material to reflect on and work with. If you like that kind of therapy, to dissect what goes on between the two of you.