I have been through the exact same thing and know how stressful it is

You can't worry about the 'what ifs' as they may never happen! And if they do happen, you will deal with it at the time. If it does become too much, maybe you need to find another line of work? I was so stressful at my last job but luckily come across a home working job which I am doing now.

What's really important here is you don't go back before you're ready!

I hope everything turns out ok for you