I don't believe that the full weight of all of this rests completely on your shoulders.
It really bites that something so uncharacteristic of you isn't having an ounce of compassion or understanding.
Maybe say, I'm not trying to get you to claim responsibility for wrong doing. That you aren't being accusatory. What you do need is some objective compassion, some way of saying he can value how upset it makes you. That you aren't looking for reassurance just some inkling to understanding why such a thing would rattle your nerves.
There's no real wrong on your part and that you value that he honestly believes he wasn't doing anything wrong. His heart for you may have been in the right place but the way the boundaries/lines blurred it was unsettling for you. Doesn't make you crazy. Dismissiveness of your feelings or your side of seeing it would drive anyone mad more so than the circumstances.