I keep starting to post this, then stopping. I'm not sure if this will help or not, but going to share it anyway, in the hopes that it could possibly make you feel better. I think your cat was rather elderly, right?

So the other day, in relation to a discussion on difficulty grieving something that's left uncertain, I happened to mention what happened with your cat. Dr. T (who has owned a number of them) said that cats are interesting in that when they sense that they're near death, their instinct is to go someplace where they can be alone. So that will often lead to them escape a house/apartment so that they can pass away in their preferred way.

Perhaps that's what happened for yours? That it wasn't you being careless and letting him out, but he was looking for a moment to escape because he knew what was coming, so he took it. And he didn't suffer (as in, wasn't wandering the streets for days), but instead went out peacefully on his own terms.