You know what is wrong -- you worry too much. It does not appear you will improve without therapy and medication. If you are not going to take what the doctor prescribes, it seems only therapy is an option. What you are talking about is having a big impact on your life. You can stay the course or get help.
Good luck.