I worry too. I hold a part-time job (barely). Social Security sent me a letter that they were reviewing my case. Then, last week, I got a letter stating that they were no longer reviewing my case and wouldn't be contacting my doctor. Strange but I'll take it.
Of course, I was IP in December and get ECT every other month, so I'm "lucky" I have that on record. I understand your worry though.