Tackling jealousy takes some finesse. OK, he has a past. His past, doesn't include you, but it's his past.

Now, look at what else, might be bothering you, right now. You mentioned that it's you doing the initiating, so there's the chance, that underneath it all, the need to feel desirable is at question. Words alone, cannot resolve that, actions will.
Your post, comes across in a way, that leaves me questioning, did he leave this out of communication, prior to your being intimate with him? That could lead to feelings of not trusting his honesty or ability to be honest. Also, could leave a sliver of resentment and betrayal.
Your post, also mentions, much talk about these former women. It would leave room for concern about his sincerity towards you. Trust needs to be there. Words alone, cannot rectify this.

Jealousy, is an interesting emotion. Usually, it's a combination of feeling certain needs are missing and also from miscommunication.

There has to be a sense of specialness, in spite of his past.

Hope you can sort through this emotion.