Hmmmmm.....

I'm not sure its him that you are having a problem with letting go. It seems more like you are having a problem with the fact that you didn't speak up in certain situations. Whether or not he was joking in some things he said is rather irrelevant to the point that it made it you uncomfortable and he shouldn't have been saying it ethier way.

You need to forgive yourself for letting him get away with controlling and trying to degrade you.

He may be a good person and does not realize that he is passive aggressive. I suggest in the future if you socialize with him again that you let him know that you don't take too kindly to some of his actions and comments and that as a friend he should respect you.

For now forgive yourself and keep your head up and don't let those negative comments and actions define you. Define yourself.

Lots of love and hugs