"I'm sorry you took it that way" is not taking responsibility for hurting your feelings. The proper thing for him to say is "I am sorry I hurt your feelings" or "I am sorry that I hurt you". That's taking responsibility. He's making you responsible for being hurt, which is called blame shifting and is a verbal/emotional abuse tactic.