He must have selective memory. You were let go from his company. You were unemployed for pretty long time after that. You then found a job. That job didn’t work out. Then they offered you job at this company again.

He made it sound that you quit the first time to pursue better pay. That’s just not what happened. I’d probably write to him that you never left to pursue more money. You were fired (laid off or whatever they called it) and you had no job, you took a job somewhere else because you had to have a job!

What happened then is not even remotely what’s happening now. He is an idiot.

He might be theoretically right that you don’t know how that other company will work out. But one never knows how jobs would work out. There’s no reason for him to repeat some trivial common sense. Every time people switch jobs, they take risks. So what. He is a weirdo.