Google has declared the phased-out retirement of the search console crawl rate limiter tool, set to be eliminated on January 8, 2024. This decision stems from improvements in crawling efficiency that have rendered the tool essentially obsolete.
Search Console Crawl Rate Limiter Tool
Introduced to the search console in 2008, the crawl rate limiter tool was devised to help publishers by offering them control over Googlebot crawling, preventing it from overburdening servers.
Previously, certain publishers faced issues with excessive crawling, leading to servers struggling to serve webpages to users. In response to numerous complaints, Google responded by integrating the tool into the search console.
The tool’s effect was twofold—it allowed publishers control while also providing Google with valuable data. Google reported that requests to limit crawling usually took around a day to become effective and remained in effect for a duration of 90 days.
The reason behind Google’s decision to remove the Rate Limiter Tool
The announcement highlighted that advancements in crawling algorithms have enabled Googlebot to autonomously detect when a server is nearing its capacity, allowing for prompt measures to reduce the crawl rate.
Additionally, Google indicated that the tool saw infrequent usage, and when used, the crawl rate was typically set at its lowest level. Going forward, the default minimum crawl rate will be lowered to a rate akin to what publishers commonly request.
According to the announcement:
“With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.
This means that we effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth.”
Making Search Console Less Complex
Eliminating the tool simplifies the search console interface by removing seldom-used features, ultimately enhancing the user experience. Publishers encountering ongoing issues with Googlebot’s crawl rate can still utilize the Googlebot report form to provide feedback to Google.
Would you like to read more about “How Does SEO and Digital PR Drive Brand Visibility” related articles? If so, we invite you to take a look at our other tech topics before you leave!
Use our Internet marketing service to help you rank on the first page of SERP.
