I don't think you're making a mountain out of a molehill. What he said to you was a very low blow. That says a lot about the kind of friend he is capable of being. It's obvious that he's willing to hone in on what he thinks is your "weak" spot and try to hurt you when he gets upset. That's not okay.

I'm all for forgiveness. I have forgiven many people for many things. I don't forget, though. People who make a point of trying to hurt me don't get a chance to do it again.