I guess you have to decide what you want? If you want a relationship, he doesn't seem like relationship material (whether or not you feel you love him). I would follow my heart and get out as it sounds like he's more into having a good time at the moment and into things you're not interested in (three-ways with gorgeous, younger women).