Tell him you have a weekly commitment and need your entire attention on that focus while you are on your lunch. Tell your boss you hope he understands and doesn't mind. 'Personal commitment' or like terms makes it sound private where he won't ask so it doesn't come ou awkward.