If you are really wanting this relationship you can attempt to talk it out with him. Remember to make a lot of "I" statements for example "I feel _________ when you ________." If he cannot discuss this with you then you must really evaluate if you want to deal with this behavior long term, it sounds to me like continuing the relationship will cause you a lot of heart ache.