maybe you can reach out to him as a friend in a different way and a way that keeps you safe (if you even want to). Maybe explain why you can't go, but offer dinner or something you do feel comfortable with. That way you have done what you can to spare the friendship, and its his to waste.