I'd say wait 3-4 days, like you said, and let it dissipate. If anyone said to me "I'm sorry" (about hurting me) and then right after "lets drop it now." I might feel invalidated too.

Maybe you guys just aren't a good match for interaction.