Never be someone's last minute plan or second fiddle. In this situation, you have made yourself to be a convenience for him and a bit of a doormat - I hate to say that, but you have. Please show yourself some self respect, cut it off, and find and be with someone who wants YOU as much as YOU want them.