It seems you are mostly worried about his feelings if you take a break or break up permanently. What about what you want the most? Concentrate on that.

Also, is he more dependent on you? Does he have friends and like to see his friends too? If he does, then it shouldn't be a problem for you to have your independence. But if he's far more dependent on you, it will be harder. Not only that, but that's not healthy.

It seems you do need to have a conversation with him about what you need and want. If he cannot accept or respect that, then you have your answer.