Your boyfriend is out of line, but the situation with this other guy sounds like bad news, too.

I would be angry if my boyfriend had some woman that had the hots for him at work, someone flirtatious enough that management was called in, and then he got a ride home from the person and didn't immediately respond to my text asking who was driving him. Not saying it's rational, but it's how I would feel.

Obviously, since you haven't cheated on him, your boyfriend is wrong -- but I would try to imagine how you would feel if the situation was reversed. Maybe if you do that, your boyfriend can do the same for you and try to imagine how irritating it is to be accused of things you aren't doing.