Yes, I would. He's not going to offer something that he doesn't want to do and feels like you need. He can take care of himself. If you feel it will be beneficial to you, then accept it without guilt.

I know with mine, I've tried to turn him down in the past, not because I didn't want to talk with him, but because I didn't want to be a burden and I felt like he needed his vacation time free of me. He's told me that he would feel better if we could talk, so that kind of gave me some room to accept his offer. We've never really done a full session though. It's more of an extended check-in. Once we tried Skype but I didn't really care for that at all. The video was all jangly and he was outside and I could hear other people and I started to wonder if they could hear me talking. Funnily, I still couldn't make eye contact even over video and I thought it would be easier.