I think you should decline any time there is any hint of them sighing or being annoyed. Why anyone would offer and sigh. That says more about them than you

Be independent..