I don’t think you’re crazy at all. It’s not quite the same, but I’ve had people who have hurt me in the past try to reconnect; I felt guilty pushing them away, even though I knew it was necessary for my own mental health. If I found out they were getting married I’d wish them well, but leave it at that.
It doesn’t sound like your old friend was very respectful of your personal wishes, if he was trying so hard to push you back into the church scene if you weren’t ready/didn’t want to return.
Bottom line, do you really want to go to the wedding, deep down? You don’t have to answer me, just listen to your gut and decide accordingly.