Being tired and in the ADD Fog is totally common. Before my diagnosis I was pretty good early in the day, if I got up and got going, but by mid-afternoon if there were no crisis at work to keep me going I'd be falling asleep off and on most of the afternoon. It was really bad when I was driving home from work (5-6pm) I'd be struggling to stay awake?!?!?

Have you ever taken a higher dose of your Adderall XR? 20mg is a fairly low dose. I take generic Adderall (non-xr) 20mg at 5am, 20mg at 10am and 20mg around 4pm during the work week.