I agree with several of the comments.

Don't ask. If they get annoyed and walk away from you it's their problem for not telling you the issue in the first place.

Continuing to think you are annoying someone can cause more annoyance and even on yourself; so don't dwell on it too much.

What helps is knowing you are worried and accept that you are because when you accept the emotion then you will sense some relief from it.