Forgiveness and trust are two entirely separate things. Forgiveness is given by your own grace, trust is earned through effort. It's not you, in this case, it's him. He has to earn your trust, and to do that he has to decide if he wants it. It sounds to me like he is being dismissive and only wants to make the effort.