You have to decide that for yourself. Don't listen to your old boss. If you want to give it a try, do it. Better to try then to always wonder about it. Disability will still be there if you need it later on, unless the government implodes anyway.
I'm about to take a temporary job myself after several months without. Very nervewracking. I understand.