YOU feel you want to give the work thing a go, and good solid reasons to pursue it! It's not up to others to decide, and I'm sorry they took such a negative approach.
Go for it!

Yeah, maybe there is stress, but maybe also you have the option of doing part time (or other ways to reduce stressful aspects) and find it manageable
because of the benefits of working with people and socializing. You can always leave if you find it unmanageable, right? So might as well try.
Keep us posted jack123, ok?