Unless you do something really bad it is quite hard to ruin someone's life. I suspect that you in fact never do anything bad on purpose.

As I see it (and I may be horribly wrong) we all have options in how we deal with others who impinge on our well being - we can walk away, we can collude in unhelpful behaviour, we can act with kindness and curiosity - how others react is not entirely up to you.

I don't think you are arrogant, but I suspect you might have less self-awareness than others which is no doubt part of your condition so not exactly your fault. In any case you are clearly working on it which is highly creditable.