That is because you're cis (as I remember you stating). You don't have to worry anyone getting your gender wrong in your family, your friends, do you? Assuming is for fools. To teach yourself to be open to critical thinking free of assumptions but based on logic and fact, you'll find this is a rather easy habit to break. I honestly don't care if you don't like that I dislike people assuming mine and others gender. You're not non-binary/trans*. You don't have to face what others and myself have to face everyday, do you? If someone calls you "sir" instead of "miss" or the other way around, does it cause you to panic? Or become dysphoric?

I think what you're struggling with is the idea that you have a certain amount of privilege over non-binary/*trans folk. Accept it, because it isn't going away. Someone in my position saying "cis-gender people HAVE privilege" is just as fine and good as me saying "someone who doesn't have a mental illness HAS privilege over me".