Maybe you should go and tell him that you were so upset that you didn't want to go. Ask him for suggestions. If you really don't think it would work out, maybe try someone else.