Did the doctor remove it and have it biopsied? It can take anywhere from a few days to a few weeks to get the results of a biopsy back.
I have the dermatologist remove suspicious skin growths (I spent time in the sun when young.) He tells me upfront if he thinks it's anything to worry about and he sends it to be biopsied. So far, he's always been correct just by looking at it. He's a specialist in skin cancer and specialists can be quite good at knowing just by looking at something.
If you're still worried and didn't have it removed, please go ahead and do it.
***
As a side note that may or may not relate to you ... great worry or fear about health issues can be a symptoms of certain types of depression. Some depressed people get sad, others get agitated, some rage, some sleep and eat too much, others can neither eat nor sleep. Depression is not just one set of one-size-fits-all symptoms. It's a complex problem.
Feeling excessive worry about health issues, especially fear that one might have a deadly disease, even after being reassured by a primary doctor and a specialist, could indicate a tendency toward depressive feeling and thinking.
Is it possible you're depressed or heading in that direction? It's just something to think about as a possibility. Take care. I wish you the best.