Yeah, please think about it. I mean its not like you're IGNORING him in any way at all. Just putting a buffer there, you know like this:

Haha, laugh about it too! I mean you're in the driver's seat in this situation, you have your head on straight, and your heart in the right place. You want it to slow down, then girl, slow it the heck down! Just meeting up with everyone at the event and skipping the dinner might give him the message.