Sadly, I agree with you. There are always going to be numbskulls that will take further advantage of you when you put your feelings out there, like bullies. They feed on making others feel bad.
You're perfectly right and addressing these people by "You." Bullies, in reality bully to make themselves feel stronger because in reality, they are very insecure personalitites. Emotionally immature people also will not respond very well to this strategy. My youngest has accused me of being self-centered "because everything is always about you." I've changed my tact with him from time to time and said "YOU said you would...!!" YOU promised me that...!" Usually, a shouting match ensues. Later I bring up the subject, we discuss the difference, and he's beginning to see the difference. But he's my son. We don't have that responsibility with others. I guess the answer is in knowing who you can discuss things with in this matter and who you can't. Still
STOP, THINK, ACT can greatly help us with our own feelings and self-worth even if we leave out the "I" statements, don't you think?
__________________
Psalm 119:105 Thy word is a lamp unto my feet, and a light unto my path.