Interesting how he did not care to talk about your past until it became important to "him".
You wanted to be up front because it was important to you, he was not interested until it was important to him.

This is important to see about him. His comments towards "you" that have been negative are about "his" needs not yours. Healthy relationships require the couple to "care' about each other's needs, not just how things are going as long as "their" needs are satisfied. You tried to be honest about yourself up front, you have been honest with him from the beginning too. You did not do anything wrong. His behavior is being "selfish". If he continues on this path even though you have drawn his attention to it being toxic, then he is not going to change and can end up turning you into a codependent wrapped up in "his needs" all the time which will include picking on you when he starts feeling insecure.