He said he "wasn't allowed to receive gifts" ?? Wasn't allowed by whom?

I've never given a gift to a therapist, but I don't see where a small token should be thought inappropriate. That it was related to some interest he expressed was, IMHO, a nice touch, as long as it was something small and not costing more than, say, $15 or thereabouts. Like if my therapist told me she had a glass menagerie, and I brought her a little glass penguin, I wouldn't think that was in any way too forward. The normal response would be, "That's very pretty. Thank you."

I think your therapist is sounding kind of neurotic. To say he now believes he "never should have told you he liked such and such" is him suggesting he needs to be in some kind of a social straight-jacket that seems like him having excessive scrupple. He sounds to me like he has some insecurity about his own professionalism.

Without knowing the gift, it's hard to evaluate this. I would advise you to keep in mind that therapists are not necessarily perfectly well-adjusted themselves. I know you admire and appreciate him. That's good. But he may some problems of his own.