I don't think you said anything wrong. I don't think toy need to apologise either.

I'm torn though, because feelings are feelings. Your pdoc was hurt, but it's absolutely not your responsibility to nurture or burden yourself with his feelings. Honestly, it sounds like he needs to work those feelings out with a supervisor, other colleague, or his own therapist.