The therapist has an obligation to current clients first. You are not out of line and it would be a positive thing to express your fears to your T. I think most good T's would refer him to someone else or otherwise decline his request for therapy if it is likely to impact on you. Don't worry - it's ok to put yourself first here!
I recommended a friend to my current T and discussed it with T, including possible problems, and I know my T only accepted to see her because he knew I was fine with it. I was more attached and insecure when I was with my first T. I wouldn't have wanted him to see anyone I know and I would have told him so.