I think some people just think their s/o don't warrant a card etc. Like familiarity breeds contempt. He asked for his gift back, the nerve ! Have you ever actually explained how his behaviour makes you feel ? You should not have to explain how his thoughtless ways effect you, but hey some people are totally away with the fairies. Make him see what it means to you. Spell it out. He's happy to receive your gift, yet simply doesn't bother to think of a gift or card for you. I wouldn't say you should ignore his birthday etc. That would just be a stale mate.
It could literally be his stupidity, and insensitivity. No harm meant. Simply , he's quite frankly, a thoughtless, and has to be said, lazy, unfeeling, and tactless individual.
I would make one huge deal out of this, now. It's gone on 20 years !
He must have redeeming qualities and I'd never suggest ending your marriage over this. There however could be other factors in your marriage that warrant attention, but if this is your only issue, and I'm not by any means making light of it. Frankly, it stinks ! Tell him ! Make it plain, you deserve at the very least a card on your birthday. Once he's got it into his ( thick ) head, maybe he'll try to change things.

I really think he thinks, it's no big deal. Afterall, he loves you etc. etc. He's got away with this behaviour for a long time. I'd make a stand now, huge talk, in a calm way. Maybe enlist a sister or someone close to make him see how his behaviour hurts you.
You say family get cards etc. On birthdays, ask him straight, why don't you. Make him see it takes a moment to think, of how to make someone feel special.
He's come to accept his actions as normal, because they are, to him.

I hope this helps. Maybe he really just doesn't 'think'.
Time to make him, think hard !