I think it probably depends on your situation. Can you afford to not be working while you sort things out with the new job? Could you do the old job for a short time while you get something else sorted? What is it about the old job that makes you feel anxious? Is there any way you could manage the anxiety better? Do you think you'd feel that way in any job, or is it specific to that particular job?

Sorry for all the questions! I don't know what you should do. I can understand the anxiety though, and understand that it's a tough decision. I hope things go well with whatever you decide.