It's not a good thing that he's so dependant on you.. and you do have every right to spend some time for yourself.
I agree that you need to talk with him as soon as possible.