I don't think your pdoc was saying he was going to abandon you. It seemed like he was trying to give you hope actually. So I think it was an overreaction on your part but considering your history it's understandable. I think it's something you can talk through with him next time you see him.