That is fine that you are scared - start imagining the worst case scenario. He will resent you. He resents you. He has resented you for a while.

Picture his resentment. You have survived picturing it? Likewise, you will survive experiencing it in reality. It is not the end of the world.