I handled the situation just as you are, isolating myself as much as possible and focussing on the task at hand. There's no point trying to fit a round peg into a square hole. You're fortunate you have a job where it is possible to lay low.

I say celebrate who you are outside of work. They are the ones missing out on getting to know a great guy and his perspective on life.