I’m so sorry this has happened to you. No, you’re not crazy, although I appreciate your current state of mind might, understandably, make you feel otherwise.
Also, don’t let him or anyone else try to blame any of this on you - you don’t control anyone’s behaviour except your own. You cannot compel another person to cheat on you! It’s on him if he really did feel there were problems and he didn’t want to try and work through them with you. Most likely all he will be able to give you are excuses, and blame shifting.