Are you on Zoloft right now or any medication? I think you said you aren't, but be careful with those because there's a tendency to feel worse on them for some people, like increased risk of suicide or depression. So make sure when you do go on medications (if you do) you pay attention to your moods even more-so.

What you are describing to me is something that I also experience at work and even in classrooms. Are you a perfectionist? Do you put pressure on yourself to be the best and do the best work and if you mess up or fail you experience a crushing blow? Anxiety can be a component of perfectionism. And since this anxiety seems to start when you enter your workplace, it sounds like that might be the issue.
Your job sounds very nice actually, I would love to just sit at a desk filing paperwork and not being bothered by customers, patients or even coworkers. So I'm glad you found a job that allows you time to yourself. Is that what you want, though? Is this job suited to your personality? Would you be happier working a more technical job where you're on your feet working with your hands, instead of sitting quietly at a desk?

The only tip I can offer you is that you need to change your mindset. You mentioned in your post that your mind is tired. That's because you don't put it down, you're playing with your mind like a toy, like a rubix cube. You need to practice stillness techniques, emptying your mind, not latching onto any thoughts that cross your mind. Focusing on one thing only - feeling good. Good thoughts, toss out any bad thoughts including the anxious ones.
It's your mind that is tiring you out - nothing else.

You might "care too much" and that's why you are feeling a crushing sense of anxiety when you enter your work. Try to go in there without a care in the world. Try to feel light and not serious. Try to forget about how many hours left of work there are and "how much longer" you can hold out. Don't even look at the clock. This is why I mentioned finding a busier technical job so that you aren't left thinking so much at a desk.