In a recent LinkedIn update, Google Analyst Gary Illyes outlined his goal for the year: to streamline web crawling efforts. This follows a Reddit discussion questioning whether Google’s crawling activity has decreased compared to previous years.
Although Illyes asserts that Google’s crawling volume remains consistent, he underscores the importance of refining scheduling algorithms and prioritizing URLs with higher relevance for crawling.
Illyes’ remark corresponds with the ongoing discourse among SEO practitioners regarding the notion of a “crawl budget,” which posits that websites must adhere to a restricted number of pages for search engines to index them effectively.
Nevertheless, the Google Search Relations team refuted this misconception in a recent podcast, elucidating Google’s approach to crawling prioritization, which considers numerous factors.
In a podcast released two weeks ago, Illyes elaborated on how Google determines the extent of its crawling activities:
“If search demand goes down, then that also correlates to the crawl limit going down.”
Though he didn’t offer a precise delineation of “search demand,” it presumably pertains to the volume of search queries from Google’s viewpoint. Put differently, a decline in searches for a specific topic might lead Google to reduce its crawling of websites relevant to that topic.
Illyes also underscored the significance of persuading search engines about the value of a website’s content, emphasizing the need to demonstrate its relevance and quality.
“If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.”
While Illyes didn’t delve into specific strategies for accomplishing this, a potential interpretation might involve ensuring that content stays aligned with user trends and remains current.
Emphasis on Content Quality
Google has previously debunked the notion of a fixed “crawl budget” as largely mythical. Instead, the search engine’s crawling determinations are fluid and influenced by the quality of content.
As articulated by Illyes:
“Scheduling is very dynamic. As soon as we get the signals back from search indexing that the quality of the content has increased across this many URLs, we would just start turning up demand.”
Illyes’ endeavor to enhance crawling efficiency through decreased crawling and data transmission aligns with a path toward a more sustainable and pragmatic web. As he actively solicits input from the community, Illyes welcomes suggestions for compelling internet drafts or standards from entities like IETF or other standards bodies that could bolster this initiative.
With Illyes’ insights, here are several actionable steps you can implement:
- Emphasize quality: Concentrate on creating top-notch, pertinent, and captivating content that meets user needs and mirrors current search trends.
- Maintain content freshness: Regularly update and enhance your content to ensure its ongoing relevance and value to your audience.
- Implement technical excellence: Establish a well-organized website structure and robust internal linking scheme to streamline crawling and indexing processes.
- Monitor search trends: Stay attuned to evolving search patterns and topics, changing your content strategy accordingly to stay pertinent and attract crawler attention.
As you refine your SEO approach, keep in mind the core insights from Illyes and the Search Relations team at Google. With these insights, you’ll be well-prepared to thrive even in scenarios where Google reduces its crawling frequency.
Would you like to read more about “Insights From Gary Illyes Regarding Google’s Crawling Priorities” related articles? If so, we invite you to take a look at our other tech topics before you leave!
Use our Internet marketing service to help you rank on the first page of SERP.
