It doesn't sound like you aren't abusing them or your workplace in any way, although they may see your tweaking your schedule as a privilege. As I am sure you know, some people love drama and the office setting is notorious for harboring people who feed on it. You won't be able to change others so maybe the change needs to be a inside job - affirm yourself for the good job you do and work on forgiveness and compassion toward your co-workers.