From what you have shared you did the right thing. You needed to have your home be your own space. It’s understandable that you don’t want to visit him right now as he may complain and act out and try to encourage you to feel embarrassed and guilty.