Sounds like a terrible boss.

Its not your fault you have a boss who doesn't have great hiring skills and makes it easy for people to switch work hours.