I think it would be fair for you to say, "You're my daughter and I love you. I will miss having you around for the holidays immensely. However, I can respect how you feel. I am a phone call away if you change your mind."
I sort of agree with your brother about pushing the envelope with you. Rebellion is more effective when it's a known sore spot. Although with sexuality there is the biology factor. but with everything else it could be a bit of spite.