Unfortunately the can deny you for pretty much whatever reason they come up with. But its utter crap they are using stuff like that against you. Therapy based activites and an active social life are two entirely different things.

Best thing to do is keep appealing, having a lawyer should help....at least from my perspective. I am not sure quite how it works but I would assume you get a different judge if you reappeal after an unfavorable decision. Also just make sure you don't downplay your symptoms at all...not saying you do but sometimes people in general are in the habit of not wanting to express quite how bad they feel.