How Google Core Updates Impact Indexing
How Google core ranking algorithm and system updates impact crawling & indexing
Indexing Insight helps you monitor Google indexing for large-scale websites with 100,000 to 1 million pages. Check out a demo of the tool below 👇.
Google ranking updates don’t just impact rankings.
Instead, the system-wide updates actively cause Google’s systems to reprioritise what is worth crawling and indexing.
In this newsletter, I'll explain how the Google core updates impact the priority of pages that Google crawls and indexes. And show you examples of this happening in the wild.
I will provide evidence that Google updates impact crawling and indexing.
Let's dive in.
🔄 What are Google Core updates?
Google core updates are significant system-wide changes that Google teams make to the search ranking algorithms and systems.
Google rolls out these updates several times a year (history of updates here).
When Google rolls out these core updates, they significantly impact the SEO traffic of websites. And people in the SEO industry talk about it A LOT.
The most recent Google update happened in December 2024 and November 2024.
Many SEO professionals focus a lot on the impact of visibility and SEO traffic on websites after a Google core update.
However, that isn’t the only impact Google core updates have on your website.
Indexing Insight shows that Google core updates impact the crawling and indexing of your pages.
🧮 Google Core updates impact indexing
Indexing Insight has identified that Google core updates impact:
Indexed pages - Google actively remove indexed pages
Not indexed pages - Google deprioritises crawling of not indexed pages
#1 Google core updates actively remove indexed pages
In the first example, the client is a large multi-lingual website (over 1 million pages).
Since the roll-out of the core updates, they have seen 144,000 pages actively removed from indexed to not indexed. This was over about 30 - 40 days.
We know they were previously indexed because of our unique ‘Crawled - previously indexed’ report.
In the second example, the publishing website is not as big, with less than 10,000 pages.
However, what is interesting is that at the end of the core update in December, the number of ‘crawled - previously indexed’ pages started to increase.
The website has had 500 indexed pages actively removed by Google’s index.
#2 Google core updates purge URLs from the index
A Google Core update can cause the indexing system to actively ‘forget’ URLs.
It seems that Google core updates delete the crawl priority of not indexed pages, especially for websites that have seen pages actively removed from Google’s index.
For example, a programmatic SEO website using Indexing Insight to monitor pages.
Since the Google core update, this website has seen not indexed pages moved from ‘crawled—not currently indexed’ to ‘URL is unknown to Google’.
Going from 5,000 pages to 10,000 pages in 30 days. After the Dec-24 core update.
A comment from Gary Illyes on LinkedIn mentions that URLs with the ‘URL is unknown to Google’ have zero priority to Googlebot.
Gary mentions that URLs can move between states as Google collects signals about a set of URLs. Over time, Google may forget a page and make it ‘unknown’.
A Google core update might actively cause the index to ‘forget’ the URL.
🧪 Evidence that Google updates impact indexing
The short answer is yes.
Research suggests that Google updates to its systems does impact indexing. And there are three big pieces of evidence to support this idea:
📚 A 9-year study on Google’s index
🚮 Google patent on how Google manages indexing
🗣 Googlers confirm pages can be suddenly removed
📚 A 9-year study on Google’s index
A 9-year study published in Feb 2016 by Scientometrics found that the estimated number of indexed pages in Google was impacted by updates to their system.
As you can see in the graph from the study below, search engine indexes don’t show monotonic growth. Instead, the index size is highly variable due to constant system updates.
However, this study was done in 2015. And A LOT has happened since then.
Even in the last 5 years, Google has launched numerous official (and unofficial) updates to its system. These include Core updates, spam updates, link updates, and many more.
These system and core ranking updates will impact the number of pages indexed in Google.
Google constantly fine-tunes its algorithm and index to ensure its users are served high-quality pages. The definition of high quality is up for interpretation, but this is Google’s goal.
🚮 Google patent describing how Google manages indexing
A Google patent titled ‘Managing URLs’ describes how Google might manage pages in its index.
The US patent describes using ‘importance thresholds’ to handle the trillions of crawled and indexed pages that commercial search engines will handle daily.
Why?
The patent describes how search engines have a finite number of pages that can be indexed.
Interestingly, this patent describes how a search engine like Google uses dynamic importance threshold scores for each URL to determine whether it should be crawled or indexed.
This process of defining the importance of each URL is continuous.
The importance thresholds help search engines, like Google, keep their indexes as high-quality as possible. Based on the importance threshold, pages are added to and removed from the index.
If new pages have a higher importance score, and there is no more space to store indexed pages, then old pages will be removed.
Note: I will write more about this patent in a future newsletter, as I am not doing nearly enough justice in this one.
🗣 Googlers confirm pages can be suddenly removed
The final piece of evidence is from the Googlers themselves.
Gary Illyes, on Twitter, has said that ‘index selection’ is tied to the quality of the content. If there is more space in the index, low-quality content will be indexed.
This comment backs up the idea of an ‘importance threshold’ score described in the patent above.
Gary also said this in an interview at the SERP Conference. He talks explicitly about how Google is deindexing a lot of URLs due to the quality of the website:
“Since February, where suddenly we just decided that we are de-indexing a vast amount of URLs on a site just because the perception, or our perception of the site has changed. And in general, also the the general quality of the site, that can matter a lot of how many of these crawled but not indexed, you see in Search Console. If the number of these URLs is very high that could hint at a general quality issues.”
- Gary Illyes, SERP Conference May 2024
Interestingly, an unconfirmed Google update was released on February 14th, 2024. Based on what Gary says in the interview and the unconfirmed update, this supports what the study found in 2016.
When Google updates its system, the number of indexed pages is impacted.
How Google updates impact crawling & indexing
Based on this evidence, it becomes obvious Google core updates impact indexing.
When Google rolls out a system or core ranking algorithm update, it must impact the ‘importance threshold’, determining whether a page is indexed.
This is why we are seeing pages actively removed from Google’s index.
However, Google core updates impact more than whether a URL is indexed. They also impact the crawl priority of your URLs. And whether the index ‘forgets’ your URLs.
So a URL can move from ‘crawled - currently not indexed’ to ‘URL is unknown to Google’.
The patent supports this, describing how the importance threshold impact whether a page is prioritised for recrawling or a new URL is crawled.
A URL index state can be reversed when Google Core updates are rolled out.
What does this mean for you (as an SEO)?
It’s important to remember three things from this post:
Google core ranking updates don’t just impact rankings
Google core updates can actively remove pages from the index
Google core updates can deprioritise the crawling of your important pages
When assessing whether a Google core update has impacted your website, don’t just look at your Search Analytics or third-party visibility graphs. Also, examine your page indexing report.
Why?
Because even though you might see a recovery or decline for your website after an update…
…it doesn’t mean the update hasn’t caused Google to deprioritise your not indexed pages as less of a priority to crawl. And it indicates bigger quality issues on the site.
If you want to analyse to see the impact of an update on your website, all you have to do is follow this quick process:
Go to the page indexing report
Filter on ‘All submitted pages’ (need to have XML Sitemaps submitted)
Scroll down to the ‘Why page aren’t indexed’
Sort by pages on the far right of the table
Check to see if you have seen an increase in ‘Crawed - currently not indexed’ pages
Check to see if you have seen an increase in Discovered - currently not indexed’ pages
Do you see an increase in the number of pages in these reports after a core update?
Yes?
There is a high chance that Google’s indexing systems are:
Actively removing your important pages from the index.
Actively deprioritising your important pages to be crawled by Googlebot.
You can read more about how index states indicate crawl priority.
📌 Summary
Google core updates impact the crawling and indexing of your pages.
In this newsletter, we’ve discussed how Google core updates and system changes can impact the number of pages indexed by Google. Indexing Insight has provided examples of core updates actively removing pages from Google’s index.
We’ve looked at evidence to support the idea that updates impact crawling and indexing.
Finally, we’ve reviewed a simple technique for identifying indexed and non-indexed pages impacted by Google updates in the Google Search Console.
📊 What is Indexing Insight?
Indexing Insight is a tool designed to help you monitor Google indexing at scale. It’s for websites with 100K—1 million pages.
A lot of the insights in this newsletter are based on building this tool.
Subscribe to learn more about Google indexing and future tool announcements.
Great article! Thanks for sharing. I didn't think about such impact of Core updates before