I take stimulants for ADHD and to fight fatigue/brain fog from my normal medications. I see you take Wellbutrin, and there's a chance that is activating enough for you to forego the stimulants.

Have you tried eliminating the Adderall and caffeine on a given day to tell the difference they make? It's unlikely they're contributing to the fatigue.