The other possibility that's been causing us problems as of late are AI crawlers or bots -- software AI makers use that crawl the web in order to help train their AI. Unlike search engine crawlers, many of the AI ones are not friendly to obeying the standards for polite crawling of websites and instead try and request every page the site has in in as little time as possible. This can result in hundreds of requests per second, which our server simply can't handle.
They come and go and we've put into place safeguards to try and prevent these bots from harming our site. But with so many of them now working, it can result in problems for us sometimes. This may also be the cause of the occasional freeze-ups.
__________________
Don't throw away your shot.
|