I don't think you are too harsh, I think you are very smart. There was no reason for him to lie to you. He chose to lie to you anyway. When someone has to hide something from their partner, that's cheating. It doesn't matter if there is sex involved or not. He hid the fact that he was meeting with an ex. There was no reason for that.

Now he is wanting to see you less than before. He is breaking things off with you. I would advise not telling him what you know. He will just lie his way out of it and/or tell you that you've misunderstood, and then he will become more secretive the next time.

Trust your instincts. Walk away. Give him the same consideration that he has given to you, which is none. Save yourself before this becomes a huge mess.