Poppy, I just wanted to give you a big hug...

You are not the first person to do someone else a favor only to have it backfire. It happens all of the time. So please don't beat yourself up over it.
I, also, have a tendency of letting my emotions think for me without going through all of the possible outcomes.

Clearly you are in a bit of a pickle, but is it possible that you have let your anxiety snowball a little? I know I do that all of the time.

I agree with the other posters that asking friends and family to be there on the day of your move is the best thing to do. They will be supportive, helpful, cheerful (when fed pizza) and will act as a buffer between you and this guy.
Just knowing that he will be at a psychological disadvantage (he is just one and you will be many) will probably comfort you a lot.