I think it might be like wanting to be invited to the party but not wanting to go... which is my general feeling about parties!

I bet it's a combination of the things you suggested. Curiosity plus missing the person plus maybe forgetting that they are annoyed when the other person is not right in front of them.