YES it's possible to still be scared. Looks like you have caller ID. If you do why not just let it ring. You don't need the hassle. Your present life sounds fulfilling with warmth and loving you didn't have as a child. Keeping your distance is Okay and healthy.
Good luck to you. Wishing you all the best.