Member Avatar for Dani

The number of known pages in Google Search Console blocked by robots.txt in the Coverage report just recently started going down. Over the course of this month, it went down from about 400K pages to 200K pages.

No changes to robots.txt file in 6+ months nor any big structural changes, 404'd pages, etc. in that amount of time either.

What would cause this number to go down on its own?

  • 3 Contributors
  • 8 Replies
  • 186 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

Short take tied to the thread: the Coverage “Blocked by robots.txt” number is a reporting/diagnostic count (Google’s record of URLs it discovered but could not fetch because robots rules prevented crawling), not a live statement that those pages suddenly became reachable or indexed. (developers.google.com)

Why the total can drop without changes on your side

  • Search Console periodically re-aggregates and reclassifies large groups of URLs (deduping, moving items between “excluded” buckets, and retiring old instances). An issue’s instances can be retired from the UI after a long quiet period, so counts fall even though robots.txt itself didn’t change. Also remember the Coverage data is an aggregated report and is intentionally slower than the per‑URL URL Inspection results, so you can see sudden-looking drops that are just reporting backfill. ()

Practical checks (quick workflow)

  1. Export the current “Blocked by robots.txt” list from GSC (UI is limited in examples; for full exports there are community tools that automate extraction). ()

  2. Compare old vs new exports to see which exact URLs were removed. Example (shell):

    sort old_blocked.txt | uniq > old_sorted.txt
sort new_blocked.txt | uniq > new_sorted.txt
comm -23 old_sorted.txt new_sorted.txt > removed.txt

    Inspect a sample from removed.txt with Search Console’s URL Inspection (check “Crawl allowed?”, last crawl time and page fetch state). ()

  3. Correlate with server logs / Crawl Stats: if Googlebot hasn’t recently attempted those URLs, the change is likely a cleanup/aging. If crawl attempts are recent but status changed, investigate redirects, canonical signals, sitemap updates, or crawl errors.

Notes and cautions

  • Don’t open robots.txt as a reflexive “fix” — if those rules are intentional (as states), leaving them is correct. If you actually want URLs removed from the index, allow crawling and use a noindex (or protect via auth); robots.txt alone can prevent a noindex from being seen. If the reduction looks purely reporting-related, monitor for a week and use URL Inspection as the authoritative check before changing site rules. (developers.google.com)

(Valid follow-ups: export+diff results or a short sample of removed URLs to validate with URL Inspection; that will confirm whether this was a UI reclassification/aging event or something that needs engineering.)

Recommended Answers

All 8 Replies

Member Avatar for rproffitt

What's the timeframe here? Why I ask is that the Verizon and other outages may be a factor and it could bounce back.

Member Avatar for Dani

What's the timeframe here? Why I ask is that the Verizon and other outages may be a factor and it could bounce back.

Nono, you're misunderstanding. This is the blocked by robots.txt coverage report. We don't want it to bounce back. Also, it has nothing to do with traffic.

Member Avatar for rproffitt

I still wonder about the timeframe here. And yes I see why you don't want it to bounce back.

That aside, I was just reading about a Google problem in April 2019 and their cache syncronization issues across their server farms which was not what was interesting but another item where they revealed about how they age the information. Could it be Google aging these entries in their system?

Member Avatar for Dani

I still wonder about the timeframe here.

Almost all on January 5th, and then small dips weekly since. Google Search Console coverage report only updates once a week or so.

Could it be Google aging these entries in their system?

I'm not sure what you're referring to. Are you referring to Google aging entries in GSC specifically, or within their index? Can you link to the article?

Member Avatar for rproffitt

The changes above are now reflected in the index coverage report so you may see new types of issues or changes in counts of issues.

So Google did note a change at https://developers.google.com/search/blog/2021/01/index-coverage-data-improvements dated Jan 11, 2021 and your date of Jan 5 is close enough to when they did roll out a change.

I don't see the blog entry about aging but the change I read at this link does look connected from what little I know.

Member Avatar for Dani

Does look to be changes made to the way they report the results, but doesn't explain how/why the number would go down. If they're excluded via robots.txt, then Google knows about them. If they knew about them yesteday, why did they forget about them today? Other sections of GSC report go back as far as 2017 when reporting pages.

commented: As to why, usually because they are keeping a lot more under the hat. +15
Member Avatar for odeskavita

Even though the Blocked by robots.txt in GSC is going down, this is a major issue that almost everyone has faced. I also have gone through this issue. Fortunately, there is a quick fix for this error. Simply change the robots.txt file at example.com/robots.txt to enable Googlebot and other search engines to crawl your pages.

Member Avatar for Dani

I think you are completely misunderstanding me.

Our robots.txt file exists for a reason. It is designed to block pages we don't want to be crawled by Googlebot. My question was entirely related to unexpected behavior in Google's search console report.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.