I would feel shaky also if my child were coming to live with me just out of prison. Maybe you are strong but you are human, also. When I know I am going to confronted with a difficult situation I do just what you are doing ... call in my supports. Friends, family, counselor, husband ... whoever can help me get through but be reasonable and honest.
One thing I learned a long time ago was that we always have options. Always. If it doesn't work out with you son, you have the right to find another arrangement.
Don't forget that you might actually enjoy his company and your time together - I tend to lose track of the positive when I am fearful.