Hey, listen to yourself is my advice.
And then wonder why so quickly you've let someone else who is controlling into your life. Perhaps some therapy may be of help? You've such self-awareness with your comment about not wanting someone controlling you again and that you're feeling vulnerable, so I really would listen to yourself and stop this relationship as it's making you feel bad.

Don't worry about the group of friends, feeling awkward for a few months is really a small price to pay if ending things will make you feel better. And as with everything, gossip, awkward feelings, they all tend to blow over fairly quickly.

Hugs.