I might be concerned that since you both decided that casual sex is okay for now. Maybe he has found someone that he could be connecting with and it not be just about casual sex anymore?

Just a thought.