True it has everything to do with self esteem. Only say it when it's necessary. When you say sorry for simple things, this gives other people who might not be nice the wrong message - that you could be easily manipulated and they have control over your emotions. All they have to do is show the slightest displeasure and you'll say sorry. Practice mindfulness and think before you react- only say it when it's necessary only.