My opinion based on life experience is, lose that guy. His response was not at all healthy and is a red flag.
I know the fear of abandonment is all consuming, but I urge you to at least think about your relationship in general in order to identify any other possible red flags.
Best of luck.