He may be conflicted with feeling his space is invaded (his sanctuary from the world, and private) with wanting companionship but with minimal interaction (talking). Is there a physical activity you could do with him? Fix or build or tinker with something? It will give you a mutual sense of togetherness without speaking, a mended or new thing which he would find useful once you have left and a little company without being intrusive. It doesn't have to be a 'guy' thing, you could cook or something else. But practical.