I don’t think there is a polite way to turn down a request to be added but simply ignoring those you don’t really want works as people often get the hint quite quickly.

Personally I never turn down an invite or a request, I accept and simply allow the contact to unfold or not at it‘s own pace.