That was unprofessional of the driver. If he starts calling you, I would report him. It sounds like you felt pressured into giving him your number, but you shouldn't be made to feel pressured. So sorry that happened.