Yeah, it is a thing. It's usually called "avoidance" or "avoidant behavior".

And if you want to socialize but have a strong fear or aversion to it as well, psychologists can refer to that as an "approach-avoidance conflict".

You can look them up, but take all of it with a grain of salt, of course.

Like the others said, starting with smaller and easier things then going from there is a good start.

If you really think you are sending out signals that make you seem less approachable or need help with skills, maybe you could see if there are groups, workshops, or counselling in your area. But you may have just done it once -- in that person's opinion -- and anxiety can blow those kind of comments out of proportion. You might just want to try re-engaging a little and see how things go first.