I would just go ahead and give a short version of that explanation.
"I've always preferred this other name, and I've finally decided to change it." Or something along those lines. You don't owe people a detailed explanation, and frankly, most people aren't that interested in the details.
I don't think most people would think it's odd or anything. Although they might keep forgetting and call you the old name for a bit.
I agree it's probably just the social anxiety causing you to stress over it, but do whichever choice is good for you.