Tough call. To play your hand and come clean out of sense of guilt for crossing that well known line of privacy can turn into quite a conundrum. In a typical sense, he more than likely will profess how it's you that can't be trusted, etc., etc., etc. You're posting here writing how this isn't something you've ever wanted to be reduced to, so I'll save that aspect as a given.
Which still leaves how to address this. I mean noone can control the behaviors of another, what's going to happen is going to happen. No matter how unpleasant that idea is

I guess perhaps take inventory of what is lacking for you in terms of the relationship itself. What exactly brought about that nagging feeling. And maybe address things from that angle? For instance, "it seems like you've been distant lately. Is there anything you would like to talk about?"

Hopefully others have suggestions that offer perspective.