I understand how hurt you feel because he hasn't replied. However, it sounds like you're setting up an ultimatum without letting him in on it--he doesn't know he needs to reply by Friday. Perhaps the most difficult, and most helpful, thing you could do is go Friday regardless of whether you have heard from him so you can work this out in person.

I do hope you hear from him sooner, though, even just a brief message.