You are very important. Your thoughts, feelings, who you are as a person, as a whole. If the thought of spending time with your family creates so much stress, you have to take medication, it is in your best interest to not see them. Caring for yourself, including your health, should be your #1 priority. Whether they get it or not. Please try to be gentle with yourself.
In order for you to get back on track, it is important that you remove yourself from the stressors that make you sick and start working on understanding why you feel sick or anxious or panicked when subjected to them.
The people that love you and care about you will understand. The ones that do not give you space and respect are the ones to stay away from.
I can completely understand what you are writing about. And there is nothing wrong with you that a little bit of T and a loving, supportive group of people cannot help resolve the symptoms. I write symptoms because the intense anxiety, fear, depriving yourself of nourishment...those are in place to help you cope. I am willing to bet that the people you feel sickest around are the same ones that got you to this point.
It is the best gift you can give yourself: peace. Take good care. Be kind to yourself and honor yourself. However you feel or what you are thinking, you are an adult. You owe no explanations other than an "I am sorry but I am not feeling well. We will have to pick this up another time."