I think it's normal and understandable to steer clear of one's own workplace when off the clock. It's nice that they offer extra discount days, at the same time, it's your workplace :\
Are there other ways to get the discount if wanted? Other locations? Employee holds while there so that you could grab and go on the way out the door or on break? Online?
Otherwise, it makes sense why you shop elsewhere.
Reminds me of something said when I was covering internal deliveries and transporting fresh baked goodies. Didn't the smell make me crave these goodies? Nope...the exact opposite!