I'm sorry the situation you describe is not one I would be able to relate to. But I noticed no one had yet replied to your post. So I thought I would.

My personal perspective on the situation you describe is that how you will react, if you go, is something you won't know until you've done it. However if you don't go, you'll never know & you may regret it after the fact. But at that point there won't be anything you can do about it. So my suggestion, for what it's worth, would be to take the risk & go.

There's a quote I'm fond of that is said to be advice the 11th century Buddhist yogini Machig Labdrön received from her teacher. It goes:

"Confess your hidden faults.
"Approach what you find repulsive.
"Help those you think you cannot help.
"Anything you are attached to, let it go.
"Go to places that scare you."

My recommendation is to go to places that scare you.