It's possible that he still feel bad about the dinner. If he is an avoidant, he trķes badly avoid any situation that can set him in a position to wake up criticisms or being rejecting.
Be patient, give him space...perhaps he is not in mood to talk, that's why he need space.
It's a very good offer the one you made him, about doing another project together. So, I think you are doing well.