Hey dear, I have gone through the exact same issue with my current boyfriend and he also had made a big deal out of it. We didn't speak properly for quite sometime and although howmuch ever I explained to him that I truly only belong to him and love him from my heart, he kept saying that my past somehow hurts him, I apologized to him several times for this because I love him so much and I don't want any problems between us. and also because I thought argument will do no good but remember here that he is supposed to understand you and your past shouldn't make things bad in your present. He must be in a hard place I understand picturing your past maybe but if he is true to you he will regard your love greater than everything else.
Hope everything turns out well
God bless.