You absolutely did nothing wrong. Sounds like he’s scapegoating you. I’m confused and concerned about where and how your sessions are being conducted. You say you are meeting him outside his office. His old office? Then where do you go? What job did he change to 3 years ago?