Have you talked about this with your SO? I think that's mandatory, to begin with.. you must be honest with him about your feelings.

What fears do you have, exactly? What do you think could happen?