Raging Vortex, good for you for setting boundaries w this person and I'm impressed that you don't feel bad (some of us might feel guilt). Because you shouldn't! That's a perfectly ok thing to do and I'd feel the same.
In term of feeling annoyed, well, that might be me too, depending on the person and friendship, and I hope you are feeling calmer now

Have a happy and safe holidays