I am going through much of the same thing at work. My boss will dump his work on me, and then leave (and go where??? who knows) for hours.
It is a terrible thing to say, but I quit caring (easy to say, hard to do).
I've always done "my job", so there are no grounds for termination.
The not caring part is really hard to do though