"part of me is also afraid of it, but I'm not sure of what" "Perhaps it's a fear of vulnerability?"
Could be, as you'd be baring it all literally and/or figuratively. There's risk in that.
Maybe for you, you need a strong emotional connection 1st before you can feel comfortable physically, "safe."