I haven't read the rest of the replies, so I apologize if someone has already said this. When I read your post, the first thing that popped out to me is that maybe you're afraid to be vulnerable, afraid of connection, afraid of getting attached because you maybe you allowed someone some form of vulnerability, connection, and attachment at some point in your life, perhaps in your childhood, and maybe that vulnerability, connection, and attachment was violated in some way that left you feeling, whether consciously or subconsciously, untrusting or afraid of feeling that closeness only to be hurt, let down, abandoned, or violated again. Violation doesn't just involve abuse, it can be any form of hurt that someone caused you.
I can totally understand why you would feel this if the above scenario is something that relates to you. For me, my therapist is always letting me know that she is ALWAYS there and available if I need her. She has a pager for after hours contact if her clients are in crisis and need her, no matter what time of the day or night it may be. I don't know how many times she told me to use it if I needed to. But, I was determined to never have to use it and confirm to myself that I was in no way attached to her or that I actually needed another human being to be there for me and to actually be open and vulnerable without the fear of being hurt like I had been in the past, especially by my former therapist, who apparently got her counseling degree from Captain Crunch.
I realized that I was punishing myself by not allowing her in, telling myself that I didn't really need her, but know that I totally needed her, or letting her think that she could have any control over how I felt, whether it be bad or good, because I was afraid that if I got close to her or felt an inkling of vulnerability or connection that she would in turn have a huge amount of control that came with the ability for her to hurt me.
BUT, (and that is a big BUT) she proved me and my irrational, yet understandable, thoughts to be VERY wrong. She proved to me that it was definitely okay to ALLOW her in, to be open and honest with her, to ALLOW her to help me, and to ALLOW her to get close to me, whether emotionally or physically (she gives the best hugs). She has been so successful at proving my irrationalities wrong, that I completely trust her 100%, now. I have no doubts that she has my best interest as her #1 priority in regards to my therapy and my well being.
BUT, (another big BUT) I had to be willing to take that risk of trusting her and to being vulnerable with her in order to make that connection I so, so wanted, but was so afraid of at the same time. You said, "Strangely enough, I actually want her to sit by me, yet I can't get past the anxiety of it all." I think (only my opinion) that your desire to be physically close to her is equal to being emotionally close and your mind is reacting with, "Alert! Alert! Physical closeness can lead to or be equal to emotional closeness and I'm not sure what kind of power I'm turning over to my therapist that she can use to hurt me."
Metaphor ahead --->

: I see it like I see drinking water. You know it's good for you, you have a desire to drink more of it, and you know you should have a satisfactory intake of it each day. But, the minute you start drinking it you know you'll have to take way too many bathroom trips than you really want to deal with, but you know if you drink it, it will do some good things for your body. So, if you stick to drinking plenty of water everyday, the trips to the bathroom get less and less as the days go by. Your body settles in and adjusts, and you're able to enjoy a healthy dose of water intake without the multiple bathroom trips.
So, the more you let your therapist in and get into the practice of being open with her and being vulnerable with her, the easier the connection will become, and you'll have less and less fearful moments of letting her be close to you. I guarantee you it will be a challenge, but it isn't impossible, and the benefits are awesome. Of course, you can have connection and vulnerability without the physical closeness, but I think for you, since the physical closeness is a barrier for you, allowing her to be physically close will allow more room and exploration for you to allow her to be emotionally close and for you to be able to trust her with your gift of closeness that you so badly want to give. It's something precious to you that you want to protect and I think by giving her that aspect of yourself it is to be considered a gift from you to her that she's earned.
One more thing, and I promise to stop being a chatterbox. You said, "Its like there is this invisible force between us that keeps
me from
her." I think in your effort to protect yourself you may be putting up this invisible force/wall, but your therapist is still on the other side, waiting for you, not budging from her place of support and just waiting for the moment you let her in, but only when you're ready.