It doesn't sound like he treats you or your family very well. Your intuition is telling you that this isn't right, and I would trust that small voice. He may have reasons for being the way he is, but it doesn't mean he should make you and your kids suffer.

I would use his move to Florida to create some emotional distance, if possible. What would you advise a friend in your situation to do?