If your superiors are happy with you, then I would question why your subordinates feel so comfortable to express their opinion of your situation. Perhaps your guilt is making you too open to allow that type of open relationship. No matter what their opinion, if your job allows you this flexibility - then I would begin pulling back from allowing open disrespect. A boss can't be overly friendly. Having been in management- I learned that the hard way. It is insubordination, and doesn't aid the company.

This is not a mess, we tend to be overly critical of ourselves because we do have to focus so much on our behavior to monitor ourselves. But try to own your self- appreciation, and discourage open contempt. If you don't know how to handle changing the atmosphere, and can't trust someone above you with your concerns - then seek training through management books. There are some really good ones out there.

"I have worked hard to be in my position, and I appreciate the opportunity." The end. No justifying yourself to your colleagues, it leaves room for questions - and if you spot light this, you may not like the results.