I can relate to how you feel about leaving while work was pouring in. Yet, if there is one thing I do know about payroll is that extra hours are often in question which can be worse than leaving at the scheduled time when the truck came in later than expected.
Personally, I struggle with the moments of not being there when shipment rolls in because I'm at my most vulnerable as that's when one of the coworkers that likes to go straight to the top about everything and anything that could go wrong and wag her little finger in my direction-with or without merit-occurs.
That's neither here nor there in your feeling bad for leaving your coworkers with a pile of work.
Again payroll. It probably wouldn't have been justifiable. Welcome but not justfiable as it's up to those scheduled to get the work done during their scheduled hours.
You did ask, so go forth and feel absolved of whatever guilt you may feel. Asking if needed to management was all you could do. Plus it shows you are at least willing.