Have you shared these feelings with your therapist? I really urge you to talk about them openly and to share your fears.
Any potential changes to the therapeutic relationship can be so hard.