The part about feeling "empty" when you aren't there would make me uncomfortable, too. There are kind of two parts to that for me, the second part being who would say that even if they felt that way?

People are not always precise with their language, though, so sometimes you have to go with your gut feeling about them.