I agree that its not your job to knock his walls down. That's his own job. If he has walls up with you then you need to just accept that he is not emotionally available to you. And there's nothing you can do to make him emotionally available to you.