If you already can't stand his kids then that's a bid red flag. My bf brought his two young boys into our relationship and it went from just us, to us having his kids here 50% of the time. It's not easy. I've locked myself in my room and cried, left the house, stayed at work late, all because I didn't want to come home and be mom for the night. It's a struggle to adjust. You're young and still growing. I'm 28 now and when I think back to age 24, wow. Definitely not ready for commitment. And this man is much older and unlikely to change. You NEED to move based on your medical condition and he doesn't sound like he wants to go anywhere. You said yourself he has baggage. I think your newfound desire to spread your wings is healthy and you should run with it. Take a break from this man and do what you need to do. If it was meant to be he'll follow