Yeah, I understand it can be pretty annoying. Unfortunately you can't really do anything, buf if he's loyal to you, you shouldn't have too many problems.