Yes, that's what I tend to think as well. Of the two, I am definitely the detached kind, although I don't consider myself completely avoidant, more fluctuating on the spectrum between avoidant and secure. One way how I actually started seeing it as problem when I was younger was how my detachment can really stress out some people that have more need for attention and emotional interaction. I would want to dismiss these situations by default and habitually, but cannot very effectively for example in a work environment. Personal relationships I have much more control and choice over and I do tend to mix with people that are similar to myself in this sense (and in many other).

I actually just went to see one of my therapists (I see them only occasionally now) to discuss this very issue. What the T said was that the way I describe my attitude, it more sounds professional than dismissive with solid boundaries, but some people just want to be taken personally a lot of the time and do not care so much about individual differences. The T suggested that I talk to these people directly about boundaries and work ethic instead of ignoring them.