Maybe say that you'd like to spend time with only them because you want to get to know them better, but when there's whole family it's impossible to do so?