Hi queenie and welcome!
Well, talk about it with him. Tell him what you're feeling and that it's one of your worries. And know that it can get better with lots of talking and with time.
If you search of the posts here using "abandon", "reject" or "transference", you'll find a lot of posts about worrying about T abandoning or rejecting. I worry still too and bring it up about every other session.
Here's a Question and Answer site I like that talks about it some, too:
www.guidetopsychology.com.
Talking about your conern here isn't being down, it's just talking with us. Glad you're here!