In a recent LinkedIn update, Google Analyst Gary Illyes shared his resolution for the year, which is to make web crawling lean. This comes amidst a Reddit discussion that queries if its crawling volume is lower this year compared to last year.
While Illyes reveals that the volume of crawling for Google has not changed, he laid emphasis on enhancing the scheduling algorithms and prioritizing those URLs to be crawled should be more relevant.
Illyes’ comment comes amidst a perennial debate among SEO professionals regarding the topic of “crawl budget,” whereby websites reportedly should maintain only a few pages for search engines to crawl them appropriately.
Google Search Relations recently set that myth to rest in one of its podcasts where it elaborated on the way it prioritizes crawling—a process considering many factors.
In a podcast released two weeks ago, Illyes elaborated on how Google determines the extent of its crawling activities:
“If search demand goes down, then that also correlates to the crawl limit going down.”
While he didn’t provide a precise definition of “search demand,” it would appear to be the volume of searches from the search engine’s perspective. In other words, if the searches about a subject decrease, then Google would decrease crawling of sites covering that subject.
Illyes also reiterated the need to show search engines that content deserves to appear on the websites and, therefore, evidence of its relevance and quality must be provided.
“If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.”
While Illyes didn’t delve into specific strategies for accomplishing this, a potential interpretation might involve ensuring that content stays aligned with user trends and remains current.
Emphasis on Content Quality
Google has previously debunked the notion of a fixed “crawl budget” as largely mythical. Instead, the search engine’s crawling determinations are fluid and influenced by the quality of content.
As articulated by Illyes:
“Scheduling is very dynamic. As soon as we get the signals back from search indexing that the quality of the content has increased across this many URLs, we would just start turning up demand.”
Illyes’ endeavor to enhance crawling efficiency through decreased crawling and data transmission aligns with a path toward a more sustainable and pragmatic web. As he actively solicits input from the community, Illyes welcomes suggestions for compelling internet drafts or standards from entities like IETF or other standards bodies that could bolster this initiative.
With Illyes’ insights, here are several actionable steps you can implement:
- Emphasize quality: Concentrate on creating top-notch, pertinent, and captivating content that meets user needs and mirrors current search trends.
- Maintain content freshness: Regularly update and enhance your content to ensure its ongoing relevance and value to your audience.
- Implement technical excellence: Establish a well-organized website structure and robust internal linking scheme to streamline crawling and indexing processes.
- Monitor search trends: Stay attuned to evolving search patterns and topics, changing your content strategy accordingly to stay pertinent and attract crawler attention.
As you refine your SEO approach, keep in mind the core insights from Illyes and the Search Relations team at Google. With these insights, you’ll be well-prepared to thrive even in scenarios where Google reduces its crawling frequency.
Would you like to read more about “Insights From Gary Illyes Regarding Google’s Crawling Priorities” related articles? If so, we invite you to take a look at our other tech topics before you leave!
Use our Internet marketing service to help you rank on the first page of SERP.
