If he asked you to come in early then presumably something had gone wrong, which is why he needed to call you in? This could be why he was in a bad mood, or he could have just woken up in a bad mood. It does not have to be about you, some people just take their bad moods out on others, it doesn't mean that they have done anything wrong.
If he said that he wasn't angry at you, or that you hadn't done anything wrong then why wouldn't you believe him? He is unlikely to be playing mind games with you, he is likely to have just been in a bad mood.

However, yelling at your supervisor is never ok. No matter how much you think they deserve it, and I am not surprised that you were sent home.

Turning up late just to get some kind of revenge is petty, will mean that other colleagues will have to be called in/stay late to cover for you and is likely to get you into a whole lot more trouble