Good quote. I know where you're coming from on the whole avoidance thing and not wanting to disagree with others. I grew up so afraid of adults even school teachers that I never called them by name. In class if I needed something I would just raise my hand until called upon and then say or ask very humbly and meekly for what ever I needed. This fear has followed me into adulthood and it is hard to overcome. For me it's about not wanting to offend anyone or "risk" offending them. A friend and I talked about it recently and he has encouraged me to say "No" anytime that I don't want to do anything that he asks of me. Sooner or later though we have to tell people that we are not here to serve them and that we have rights too. The reason I say that is because there are some people who will detect this weakness in us and try to use it to bully us into doing their bidding. Sounds to me like the person you work with and are trying to avoid has figured this out about you and using it to try to take advantage. It's scary to stand up to bullies at first but, it's also necessary so that we don't bottle up these feelings inside and then go off on someone when we can't take it anymore.
I've done this more than once because I almost always place others ahead of myself and that's not fair.
You owe these people nothing except to treat them like human beings. Sometimes you just have to calm yourself and get in that frame of mind where you aren't going to take any crap then, look them in the eye and calmly and slowly tell them that you aren't going to be swayed or bullied. It's very important here that you let your eyes do the talking as well and if you have to narrow them with some aggression I say do it because that says "You don't want to mess with me." If they wont back down after that then, you should take it to a supervisor and tell them that you feel that you're being harassed by this overbearing person and that you are getting stressed as a result. Don't feel bad about 'snitching' if it seems to be the only alternative. There is nothing cowardly about seeking help from someone in a position of authority. I hope that everything goes well for you and remember that you owe no one nothing. It's YOUR life after all. I believe in you. You can do this.