If you want to have a healthy relationship you will have to just put his past aside otherwise it will sabotage it. I say as long as he's honest and faithful to you and there is no reason not to trust him then just let it go.