Mueller asked about lag in Google Search Console indexing report

Google’s John Mueller answered a question in a Google Office Hours Hangout on a search console, but where the URLs were listed as excluded, but when the URL is examined, the webpage was listed as indexed.

Google’s John Mueller said he had seen reports of the anomaly and had an idea of ​​what it might be.

Why was the page crawled but not indexed?

Someone asked about an issue where Google reports that pages are not indexed, but when they are reviewed another report says they are indexed.

This problem makes it difficult for the person to accurately track the crawling and indexing statistics of the site.

The person asking the question explains the problem:

“We have like a very large number of Unindexed crawled pages listed under Excluded.

But then when we click on it, most of them seem to have been converted to indexed pages.

We are therefore really unable to accurately track the impact of improvements made to our site on indexed pages.

And I was curious, I guess, of the timeline of that.

We are concerned that this will have an impact on our crawling budget.


Continue reading below

Impact on exploration budget

The person asking the question was concerned that the crawl but not indexing error was causing a problem with their crawl budget.

A crawl budget is the number of URLs that Google assigns to crawling a site.

The crawl budget is calculated in part on the capacity of the servers to serve pages. This is called the Exploration capacity limit.

If a server is having difficulty serving pages, Google can limit its number of crawls so as not to affect the server’s ability to serve pages.

But if a server responds quickly and can easily handle GoogleBot’s request for more pages, Google may decide to increase the crawl budget and crawl more pages.

The crawl budget is also influenced by how often a site is updated.

A site that is infrequently updated may be crawled less often than a site that is constantly updated.

What was going on, which the person later revealed, is that the site has hundreds of thousands of pages.


Continue reading below

But Google was only indexing around 2,000 pages per day, which means a lot of pages were not crawled at all.

The underlying concern that has yet to be raised was actually why these other pages are not being indexed and whether this unindexed indexing issue had anything to do with the crawling issue.

But this question had not yet been asked.

John Mueller therefore only answered, at this stage, the question that was put to him, which concerned the crawled but not indexed problem and whether this impacted Google’s crawl budget.

John Mueller addressed the crawl budget issue:

“I doubt that affects your creepy budget… by the way.”

Google Crawled – Currently not indexed

Google’s Mueller went on to explain why Google could show that a page has been crawled but not indexed but in fact indexed.

Response from Mueller:

“This is something where I recently saw threads like this on Twitter where people saw URLs that were flagged as not being indexed in Search Console.

And then when you check them individually, they’re actually indexed.

I don’t know exactly what’s going on there yet.

I suspect it’s more a matter of timing as we show them in the Search Console report and then they get indexed over time.

… Then at some point, they would drop the relationship again.

And for some reason, this kind of quitting takes a little longer than it should.

It’s kind of my guess there.

Check the index coverage issue

Mueller then suggested a way to check if what was reported in Google Search Console was a real index coverage issue or if it was just a delay in reporting.

John Mueller suggested:

“One way to check this is to see if these pages actually show up for normal searches.

So take a few words from the page, research them.

And if they show up, I think there is really nothing to be done.

It’s just a report that’s kind of lagging behind.

Delay in reporting on index coverage

There seems to be a lag in the indexation ratio. Hopefully, the lag is something Google might look into in the near future as it presents a bad user experience for providing false information.


Read the explanation from Google’s developer page on GoogleBot’s crawl budget:

Large Site Owner’s Guide to Managing Your Exploration Budget


Continue reading below

Watch John Mueller answer the question about the Google Search Console index report delay.

Watch it at 22:43 minutes:

Previous Expenditure Tracking Software Market Outlook by Large Company, Regions, Type, Application and Segment Forecast, 2021-2027 - The Manomet Current
Next How to Close the Digital Divide in the U.S.