If you want to go home then go. What happened between you two is nobody else's business and it is not necessary for you to speak about it if you don't want to. You real friends know who you are and will not believe the gossip anyway. Those who don't really know you will learn soon enough what sort of person you are.