I think you are on the right mind track.

If you and your husband are able to move, do it. Then just visit during holidays and text/call them back every once in a while. That way if you start a family you don't have to worry about your children being around that much negativity.

You're not abandoning them if you're moving on in YOUR life. It's not like you intend to shut them off completely. They need to respect you and your wishes.

It doesn't make you a bad person. It makes you stronger for wanting a life with more positivity in it.