Wow, ok, RED FLAG. His comment that he could have had sex with three different women, but came home to you instead. WTF is that about? What, you're supposed to be grateful? Why would he even say this to you, and what prompted him to say this? What is the context in which he made this comment?

I would be wary at this stage, honestly. That comment would make me think he is either cheating already or is thinking of cheating.

And I agree - your gut instinct is right on. There's something fishy going on with this change in behavior regarding his phone. Perhaps he WANTS you to look and perhaps he WANTS to either get caught, or to be able to blame you for snooping and cause a fight or a breakup. Either way, it's fishy and it's manipulative.

I don't like the sounds of this. It wreaks of something that is amiss and off in the relationship.