Google has expanded its official list of crawlers by introducing details about a previously lesser-known crawler that publishers have sporadically encountered without any existing documentation.
Although Google has now provided official documentation for this crawler, there appears to be a need for further clarification based on the information provided.
Special Case Crawlers
Google uses various types of crawlers, also referred to as bots and spiders.
These crawlers can be categorized into different types:
Common Crawlers
- Primarily used for indexing diverse content.
- Some common crawlers serve purposes such as search testing tools, internal Google product team use, and crawling related to AI.
User-Triggered Fetchers
- Bots activated by users are used for tasks like fetching feeds or site verification.
Special-Case Crawlers
- Designed for specific situations, such as mobile ads, webpage quality checks, or delivering push notification messages via Google APIs.
- These bots do not adhere to global user agent directives in robots.txt, signified by the asterisk (*).
The recent addition to the crawler documentation pertains to the Google Safety user agent. Although the crawler itself is not new, the documentation is newly provided.
Google-Safety Crawler
The Google Safety Crawler is a specific user-agent used by Google for the purpose of identifying and addressing issues related to malware. It is categorized under special-case crawlers, distinct from common crawlers used for general content indexing.
Notably, the Google Safety Crawler operates independently of directives in robots.txt, making it unique among special-case crawlers. To access detailed information and documentation about the Google Safety Crawler, one can refer to the relevant section on the Google Search Central page dedicated to crawlers.
The documentation for the Google Safety Crawler, found under Special-Case Crawlers, reveals its usage in Google’s processes for identifying malware.
Noteworthy among special-case crawlers, the Google-Safety Crawler completely disregards all directives in robots.txt.
You may also like: How to Optimize Crawl Budget for SEO
This recent documentation sheds light on the Google Safety Crawler and its functionalities.
“The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.
This user agent ignores robots.txt rules.”
The full agent string for the crawler:
"Google-Safety"
Explore the updated documentation for the Google Safety user agent on the Google Search Central page within the section dedicated to special-case crawlers.
Would you like to read more about “Google Safety Crawler” related articles? If so, we invite you to take a look at our other tech topics before you leave!
Use our Internet marketing service to help you rank on the first page of SERP.
