Yes, it was rape. No, it was not your fault. No, he doesn't get a "pass" due to alcohol consumption, and he doesn't get a "pass" because you didn't get into a confrontation over it.
That is the lie that is told. If you didn't fight, you must have been ok with it and no harm done. That is a total lie. You are not guilty for being passive. It is a common response of self- preservation and confusion over an assualt like that. It is one that few understand until they go through it themselves.
However, confronting him now will probably not be fruitful. He will likely deny everything, as he indicated by claiming he didn't remember the previous night. Stay away from this person, as he clearly doesn't respect you and doesn't deserve your company.
Depending on how close you are with them, you might want to warn the other two girls that might end up in that situation so they can better protect themselves. You don't have to say you were raped, but that he tried to get fresh with you when he thought you were sleeping and you had to leave the room. Let them know he is not trustworthy. I say this with some hesitance, however, because it could make the situation more difficult, depending on the circumstances and relationships within your group.
Take away from this that you do need to be more forceful with boundaries when your safety might be compromised. Keeping the boundaries clear will empower you, and you will be able to expose those who commit a crime like this. It is not fair that we are not taught how to handle being attacked. It is not something we spontaneously know how to deal with.
I am sorry this happened to you. It happened to me, and I still don't know what the right thing is, except I learned how to better avoid it in the future by staying out of situations where I would be alone with a man that is showing bad intentions. It doesn't make it right, but nothing really would. You are not alone.