Maybe it's not about you. Maybe he has something going on in his life or body that made him work out less often or switch his gym time. When we make someone the object of our attention, we tend to think that their actions are a reflection on us, when in fact, it is about their individual choices and not about us at all.