Bing and Yandex launch IndexNow for the immediate indexing of pages
It is called IndexNow and it is an evolutionary indexing protocol that can radically change the way content is indexed by all search engines: Microsoft Bing and Yandex are the ones launching it, which, however, have also opened to the participation of other search engines, and its main feature lies in the possibility for webmasters and site owners to submit a page for immediate inclusion in the index, with the aim of offering each user the maximum benefit in terms of speed and efficiency of indexing, scanning load management and freshness of content for researchers.
What is IndexNow
IndexNow is a protocol licensed under the terms of Attribution-Sharealike Creative Commons in which any search engine can participate (with a significant presence in at least one world market): its goal is to allow site owners and webmasters to instantly submit and index their pages and content on the search engine.
Indexnow will be active from November 2021 and is currently supported by Microsoft Bing and Yandex, the two search engines that promoted the initiative and participate fully, but others are invited to adopt this open protocol.
In addition, Microsoft has announced that it is encouraging all Web Content Management Systems to adopt IndexNow “to help its users immediately index the site’s latest content and minimize the scanning load on their sites”: Wix, Duda and others plan to soon integrate this API while WordPress for the niche moment. Even CDNs like Cloudflare and Akamai work with the Indexnow protocol, as well as SEO tools like Botify, Oncrawl and others, while large companies such as “eBay, Linkedin, MSN, Github and Bizapedia” are planning to migrate to Indexnow from the Bing URL sending API.
This list lacks Google, which (according to rumors) is aware of the Indexnow initiative and received an invitation to participate, but which has not officially joined nor provided details; In fact, Google is always very strict about its application indexing APIs, only used now for job offers and livestream content.
How the protocol for the immediate indexing work
As you can read from the official project page, in its simplest form Indexnow is nothing more than “a ping to alert search engines that a URL and its content have been added, updated or deleted“; thanks to this report, search engines can quickly apply the change to their search results.
Currently (and without such a system), search engines can take days or weeks to find that the content has been modified, because they do not often search by indexing; For example, we know that Google’s indexing times range from a few hours to several weeks depending on the type of site and content, and this can lead to consequential potential loss of traffic, customers and even sales.
Another important feature, the sending of a notification through IndexNow extends the reporting to all participating search engines, which are informed immediately of the modification of Urls and can give priority to the search for indexing of such Urls, thus limiting regular crawling only to finding new content.
In the usage notes, however, it is stressed that the use of IndexNow “does not guarantee that Web pages will be subjected to search for indexing or indexed by search engines”.
What is IndexNow for
The new protocol should meet the needs of search engines and site owners at the same time, as explained in a post by Microsoft Bing.
On the one hand, in fact, it can help to make “Internet more efficient“, reducing the dependence on spiders that have to go out on the Web and scan every URL they find, and on the other hand give way to website owners to provide a clear signal that helps search engines to prioritize scanning for these Urls, “limiting the need for exploratory scanning to check if content has changed”. At the same time, then, they lighten the load of the server by bots that constantly scan web pages, allowing the server to optimally function without the additional burden of the redundant web page service that search engines have already stored.
In a nutshell, IndexNow can reduce the time for the discovery and indexing of content, but also help to limit the energy demand required for scanning and indexing operations, thus limiting the share of emissions contributing to global warming.
In the intentions of Microsoft, in the not too distant future “search engines intend to limit the crawling of websites that adopt IndexNow”.
How can we use IndexNow to index content
Generally, this article on the Search Engine Journal reminds us, search engines have two ways to get web page data: pull and push.
The Pull system is when a crawler of a search engine visits a site to request web pages and “extracts” data from the server, and is the method in which search engines traditionally work.
What Indexnow does is change the discovery of content with the Push method, in which a CMS (like WordPress, if it will provide integration) communicates to search engines when the content has been published or updated.
The adoption of the IndexNow API is very simple for developers, who must “only” generate a key supported by the protocol using the appropriate tool, host the key in a text file (named with the value of the key) in the root of the site and start sending URLs when they are added, updated or deleted. Sending a URL “is as easy as sending a simple HTTP request containing the modified URL and key,” they explain from Microsoft.
Non-developers can rely on the various platforms that have already adopted the system or will do so in the near future, while from the point of view of the individual site there are various systems to use the API.
The most useful is surely the integration in the CMS through native function, which allows each publisher to access the new protocol immediately; also many CDNs, as said, have already joined the initiative and proactively send signals on which scans could produce new content and which not, and in a similar way are moving some of the ” major players in the SEO sector”.
