Yes - if he has started to just run late - it could easily be 3 times in a row. He could have some new thing in his life causing his lateness or some other client making the day run late. I am not saying that being late would not be upsetting - I am saying it may have nothing at all to do with that particular client.
I would tell him that it was upsetting and if it bothered me enough - I would try to pick a new day and time. I would not try to guess the other client's intentions or mindset about it.