Hello worrist: I don't know how you get past this. From what I've heard & read about this sort of thing it is sort-of endemic nowadays. Almost everyone feels they have to be tethered to their phones 24 / 7. They're even working while they're on vacation... that is if they take any vacation time at all!
Many years ago, I worked with a guy who was fond of saying: "If you don't use it, it isn't a benefit." Perhaps that's just the way one has to look at it. But I know there's always that little voice in the back of your head that's saying... "you'll be sorry..."

Hope you find a way to get beyond this dilemma.