John Mueller from Google addressed a Reddit query regarding the absence of a commonly used robots meta tag and its potential implications. While Mueller’s response is logical and well-documented, it might still catch many publishers and SEO professionals off guard.
What Is Meta Tag?
A meta tag is an HTML tag that provides information about a webpage. Meta tags are placed in the head section of an HTML document and are not visible to site visitors when they view the page in a browser. Instead, they serve to communicate metadata to search engines and other web services.
Meta tags, including the meta description element, serve to convey machine-readable metadata that search engine crawlers, like Googlebot, can interpret. The Robots Meta Element, however, stands out as it holds the capability to influence the behavior of search engine crawlers.
The information transmitted via the robots meta tag is termed a directive, signifying that robot crawlers are bound to adhere to the instructions specified within the robots meta tag. Although various directives can be conveyed through the robots meta, the focus of John Mueller’s response centers on a specific meta tag relevant to the query – the noindex, nofollow meta tag.
<meta name=”robots” content=”noindex, nofollow”>
The meta tag mentioned above instructs search engine crawlers not to index the content of the webpage and to abstain from following any links. One of the frequently used meta tags is the following, directing search engines to index the content and follow all links:
<meta name=”robots” content=”index, follow”>
Despite the prevalence of the aforementioned meta tag, there exists a considerable amount of confusion surrounding it. Some individuals argue that since Google endorses the nofollow attribute, it must also endorse the follow directive.
Several reputable websites assert that Google utilizes the meta robots index, follow meta tag. However, John Mueller clarifies in his response that this is not the actual manner in which Google interprets these directives. The inquiry raised on Reddit was: “What is the impact of omitting the meta robots index tag?”
“I’m a bit confused with a website I’m working on. So, this is what the meta snippets on most of the websites I work on look like: <meta name=’robots’ content=’index, follow …. But, on the website at hand, it’s missing the ‘index’ tag. My question is: What’s the effect of the site missing the ‘index’ tag.”
John Mueller responded:
“The “index” robots meta tag has no function (at least in Google) – it’s completely ignored. Also “follow”. Google has https://developers.google.com/search/docs/crawling-indexing/special-tags & https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag to document the meta tags that have functions. You can use anything else, it’ll be ignored. <meta name=”robots” content=”topranking bestcheese”> is an option, if you want to throw people off.”
The rationale behind Google overlooking the robots index and follow meta tag is simple—it is because indexing and following are inherent defaults. Search engine robots naturally engage in indexing content and following links; they don’t require explicit instructions to perform these actions as it aligns with their inherent purpose. Google’s documentation on robots tags provides guidance on this matter:
“The default values are index, follow and don’t need to be specified.”
You can find the comprehensive list of accepted directives for Google. See the list here.
If the specific robots meta directive you intend to use isn’t included in that list, Googlebot will disregard it. Is the Index, Follow Combination Entirely Ineffective? Indeed, it is established and officially recognized that, in the case of Googlebot, <meta name=”robots” content=”index, follow”> serves no purpose and is overlooked by Googlebot, making it an extraneous use of HTML space.
Bing handles index, follow similarly but with a slight variation, as outlined in the official Bing documentation for meta tags.
Google Explains How Subdomains Contributed to the Perceived Indexing Problem
According to Bing’s documentation, here is what is mentioned about the index directive:
“By default we assume “index”, but if needed you can use <meta name=”robots” content=”index”> to explicitly state that we may index the page.”
And this is what it says about the follow directive:
“By default we assume “follow”, but you can explicitly state “follow” if so desired.”
The underlying rationale is that optimization aims to streamline the process for search engines to index and comprehend content, necessitating the elimination of any elements that could hinder this objective.
In this particular scenario, it is highly probable that the absence of a meta-description will not significantly impact the outcome. However, there is another aspect where the directives “follow” and “index” can be a source of confusion. Certain publishers use the following robots meta tag:
<meta name=”robots” content=”noindex, follow”>
Certain websites suggest that using the “follow” directive will prompt search engines to follow the links when a page isn’t indexed. However, this assertion is inaccurate due to the fundamental fact that Google cannot follow a link on a page that is not indexed. If a page is excluded from the index, the links on that page are likewise excluded from the index.
Would you like to read more about “Google on the indexing and follow meta tag” related articles? If so, we invite you to take a look at our other tech topics before you leave!
Use our Internet marketing service to help you rank on the first page of SERP.
