I had a pdoc that would keep taking the diagnosis off of me everytime it was placed back on. Finally i had a psychologist who "put her foot down" and told the pdoc she had higher ground on the diagnosis, and that it was going to stay.

But through this, i had things such as agoraphobia placed on. which seems kind of simliar to what the pdoc is doing in your case. she/he may be focusing on only a few symptoms, and not taking a look at the bigger picture. It's also possible that the pdoc isn't knowledgeable in autism.