The one thing that stood out to me was that you pointed out that it was a short notice invite. Is it something you regret not doing or is it something else? Does this person typically make last minute plans? Are they prone to not consider you for other events when you decline? Did it make you feel like an afterthought?

You need more than same day invites, there's nothing wrong with that.