"if they wish to believe the worst then that is saying more of where they're at than anything about (me)..."

Yep

my thoughts about others believing the worst about me!

some can empathise and some can't... like you said, that is "life"