No it likely isn't fair but you need to stand back and look at the situation from another's eyes and viewpoint. Ask yourself, at what point does my activity become bothersome?