You are probably fine. There is a risk, but it is tiny.

And I think you are right that this is sort of ambiguous - declining an interview is not the same as declining a job.