Can you try your first few sessions with him? Then if you really seem to need him, bring him with you after that. Maybe with this transition you are ready to recognize that the comforting he gives you ultimately comes from inside of you and is always available to you with or without having a stuff animal in your hand.