I'm so sorry. It's cruel of him to say those type of things, knowing it will hurt your feelings. Most everyone in my husband's family is capable of verbally hitting below the belt. They say it to be manipulative, to wound the other person.
. It's not fun to be around someone like that, it damages any relationship, even when there's still an upside.
Please realize that it's not about you or anything you do. Frustrated, immature people fling comments like that around carelessly. Mature people address problems directly and without pain. It's taken me way too long to understand this. Good boundaries are your best friend with someone like this.
Hope you are doing okay. ((((Hugs))))