I think you're right, he was so young he didn't really know what he was doing. I must admit, I did not know that children of 5 or 6 could become aroused so much that they act out on it - it seems unusual.

I do think tho that he needs to see a therapist - he has a few issues to work on and I don't think that he will easily have a healthy relationship until he works on those issues. You already hit the nail on the head that his low self esteem is effecting your relationship so early on. Has he ever sought therapy?

Nobody will say leave him, but if his self-esteem becomes such a problem that you are always feeling wrong-footed, walking on eggshells or too many periods of sadness because of it, then you would be right to leave him. This is something he needs to work on, and whereas it would be good to support him when he gets help, you shouldn't take a bad relationship on if he doesn't take responsibility of his ways and get help.

Good luck.