Yes I agree. If a conversation with him is possible, I would say its the next step, since it's causing a problem in your relationship. If he doesn't have a problem, he'll probably be very understanding.