At least three things coulds be happening with the anxiety-letting loose thing.

1. It could be PTSD. If you have unconscious trauma your mind will naturally not want to let go of controlling the moment. If you control the moment, you control whether or not your mind wanders toward things that trigger the trauma. If you let your mind go free it might very well wander into a psychic land mine, so your brain may have come to associate letting loose with anxiety. Our minds want to avoid pain, but to do so any pain we encounter must be retroactively linked to the sequence that caused the pain. If,for instance, you stub your toe on your front step your brain can only ensure you don't feel the pain of a stubbed toe again by associating the pain with swinging your foot too low over the step. Same goes with accidentally triggering subconscious trauma. Say your trauma was a car accident, and you're just idly wandering in thought thinking about bath tubs, then rubber ducks, then ducks crossing a road, then cars, then the crash... Your brain wants to avoid reliving the crash because it's painful, so it retroactively associates fear with the causal chain that led to reliving the crash, and you now have anxiety linked to ducks and baths and whatnot. In this way trauma spreads out and infects your mind like a sprawling web of sensitive associative tendrils and leads to a mind incapable of letting go for fear of facing pain.

2. Internalised hyper-critical influences: say your parents are very critical of everything you do. You naturally don't want to give them reason to come down on you, so you create your own internal critic that can stop you doing anything your parents might criticise you for before you do it, but that means you have to inspect every impulse prior to letting it go. You can never be free. The impulse feels good to pursue, so breaking from it requires an emotional motivation greater than the impulse itself, and that emotion is anxiety. As the impulse arises so too does fear, and if the fear is greater the effect will be to desist from the impulse - fear always encourages a retraction from living. Letting go means you are allowing the accompanying fear to escalate. Hence anxiety.

3. Hyper-sensitivity: you might be a very sensitive person. You can become so via genetics, prenatal stress, and even trauma which pre-sensitizes the psychic system in the same way that a wound will make the surrounding tissue more sensitive even though it's not specifically injured. If you're overly sensitive you're prone to extreme experiences. Letting go might represent the threat of overwhelm, and it has happened often enough that you are conditioned, in a Pavlovian sense, to associate letting go with the fear of overwhelm.

There could be other things also, but these three sprang most readily to mind.