I think you are doing fine with your kitty. My daughter has two cats and they have moments when they bug her too. Her cats are both indoor cats and they are fine with that. It is safer for cats to be inside IMHO, often they can get in trouble if let out and some often fall prey to coyotes too. Your kitty will get used to whatever routine you can manage, you love him, he is loved and cared for, don't you worry.