I hear you when you say you don't have "evidence". People are pretty good at covering their tracks.

I long suspected that my ex husband was having an affair. There were lots of red flags - but no evidence, per say. And my gut feelings for a year / two years proved itself right. I always say trust your gut.

But it entirely depends on what red flags you are picking up? What makes you think that he has cheated? Sometimes there are plausible explanations to the strangest things.