Yes, it can happen and it is ok to limit contact as needed.
One time I phoned a friend and asked if she could meet for coffee, she said yes and we talked a little longer then I had to say, i can't go now I am too tired.

She gets off the phone, here husband askes who was that?
she said G! whe just invited me for coffee and then cancelled. This amused them.

In the long run it is better to miss a few coffee dates and conversations to use what energy you have for healing.
Take care.
g