Sounds like he's putting himself first, and you probably need to put yourself first as well in this case. Your needs aren't being met and you have every right to end it.