I think you are allowing your life to evolve around this man way too much (it seems to be an ongoing theme of your relationship and possibly your previous relationships too).

Why don’t you just go to gym when you feel like it? Being in a relationship doesn’t mean you Must be attached 24/7 or neglect your needs to keep other person happy at all times. You diet and well being and exercise suffer because of your choice to be with him instead you can reach healthier balance if you don’t put men first at all times.

It’s normal to make each other a priority. But it seems that you are making him a priority at all times and to detriment to your own well being. While he makes you a priority only when it suits his schedule. This imbalance probably makes you feel insecure.