Welcome to Indexing Insight
A newsletter to learn more about Google crawling and indexing.
Hi, I’m Adam.
I’ve been an SEO professional since 2011.
I’ve been fascinated by a particular problem: Monitoring Google indexing at scale.
Google indexing has become a growing problem over the years. Many companies and website owners find indexing pages or keeping them in the index more difficult.
“Anecdotally, since November 2021, I’ve also seen more and more indexing fluctuations across websites of all sizes – with the greater impacts being witnessed on websites 100k, and 100-million URL plus.” - Dan Taylor, August 2022
OR
“But in recent months, we’ve seen complaints from the owners of much smaller websites and our own data shows a similar trend spreading to 300-3,000 page websites as well. In fact, client Search Console accounts show a marked change in the way Google’s interacting with content.” - Alex Bussey, July 2024
OR
“I checked a large sample of the most popular e-commerce stores in the US for indexing issues. I discovered that, on average, 15% of their indexable product pages cannot be found on Google.” - Tomek Rudzki, November 2021
I’ve seen Google indexing problems for both clients and my personal websites. It was Google indexing issues that caused me to start investigating indexing at scale.
At the end of 2023, I noticed an important traffic-driving page for The SEO Sprint was suddenly marked as Not Indexed in Google Search Console.
There was no warning, no alert. One day, it was indexed. And the next, it wasn’t.
The lack of warning from Google bothered me. So, I set out to try to solve it.
At the start of Q1 in 2024, I teamed up with a full-stack dev to create an index monitoring prototype to answer the following question:
Is it possible to monitor Google indexing of important pages at scale?
What is crazy is that we did it. We figured out how to monitor Google indexing at scale for site owners who want to check 100K pages.
You can read about the details of the index monitoring prototype on my blog.
But we didn’t stop there. I was curious if we could push the solution to its limits.
So, we’ve been working with a handful of alpha testers with massive websites to stress test our solution with 100K - 1 million page websites.
For example, we’re inspecting over 1 million important pages for a client in 16 days.
The result of monitoring and analyzing millions of pages in Google’s index in 2024?
It has broken my mental model of how Google crawling and indexing works. And how page indexing reports work in Google Search Console.
In fact, I think the daily index monitoring has helped me better understand Google’s web search architecture. And has helped understand why there are Google's crawling and indexing fluctuations.
This technical knowledge has helped me work with tech teams and debug/fix issues. I think it can also help other SEO professionals solve indexing issues.
So, the goal of the newsletter is simple:
To help SEO professionals learn more about how Google Indexing works AND help provide practical tips to fix indexing issues.
For example, the newsletter will help answer questions like:
How does page indexing status in GSC map to crawling & indexing architecture?
How does “crawled - currently not indexed” actually work?
How does Google decide what to index and not index?
How to fix “Crawled - currently not indexed”?
How to fix “URL is Unknown to Google”?
Sounds like your cup of tea? Sign-up here:
p.s. The tool to help monitor Google indexing at scale will be released soon. We’re just in the process of polishing it up.