Ahhh, yeah, I would do the same thing in that case: not want to go but feel sort of bad about it.

I recently went out with a former coworker recently who was so positive about how much better everything was there that I'm not going to see her again!