There's absolutely nothing selfish in what you're doing. In fact it's quite the opposite - you're not taking enough care of yourself.