Member Avatar for Dani

I am trying to remove an entire folder of thin content from Google to help me recover from a Panda/EAT-related penalty. I want to keep the content on the site for the benefit of users, but not waste crawl budget or have Google think that we have so many pages of thin content.

I added the folder to robots.txt quite a few months ago. While some pages are showing up as "Blocked by robots.txt", the majority of pages now show up in my coverage report as "Indexed, though blocked by robots.txt". About 2 months ago, I submitted a removal request for all URLs that begin with the prefix, but there's been no change. Google Search Console's report updates every few days, but the number of URLs that say, "Indexed, though blocked by robots.txt" is increasing, even months after the removal request for those same pages.

  • 8 Contributors
  • 10 Replies
  • 686 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by HelgaRaymond

Dani AI

Generated

Short diagnosis anchored to ’s symptoms: robots.txt prevents Googlebot from fetching the pages, so Google can’t see any on‑page or header noindex signals and instead may still index the URL based on links (that’s why Search Console reports “Indexed, though blocked by robots.txt”). The only reliable ways to remove live pages from Google are page-level controls (meta robots or X‑Robots‑Tag) or removing/access‑protecting the content; robots.txt alone won’t guarantee deindexing. (developers.google.com)

Practical, low‑risk plan (prioritized)

  1. Temporarily allow crawling of that folder (remove the Disallow).
  2. Apply a site/server level noindex so Google will see it on next fetch — for many files use an X‑Robots‑Tag header (examples below).
  3. Force recrawl for representative URLs (URL Inspection → Request indexing) and submit/refresh the sitemap for bulk coverage.
  4. Monitor Coverage until the “Indexed, though blocked…” count drops; once pages are removed you can decide whether to reintroduce stricter robots rules or keep the noindex. For urgent hiding, use the Search Console Removals tool (this is temporary — ~six months) while you implement the long‑term fix. (developers.google.com)

Code snippets (server header approach)

# NGINX - apply to folder
location /thin-folder/ {
  add_header X-Robots-Tag "noindex, nofollow";
}
# Apache (.htaccess)
<IfModule mod_headers.c>
  <FilesMatch "\.(html|php)$">
    Header set X-Robots-Tag "noindex, nofollow"
  </FilesMatch>
</IfModule>

(Headers must be served with a 200 so Google can read them; don’t leave the folder blocked in robots.txt while doing this.) (developers.google.com)

Troubleshooting checklist and next steps

  • Inspect a few representative URLs with GSC’s URL Inspection to confirm “Crawl allowed: Yes” and that the noindex/header is visible.
  • Use Search Console Links report to remove or change any internal links that point to those parameter/faceted URLs (reduces discovery).
  • If you cannot allow crawling for policy/technical reasons, consider returning 410 (gone) or password‑protecting the folder — both cause removal, but they change user access. Track progress in Coverage and only re‑block with robots.txt once you’ve confirmed deindexing. (developers.google.com)

Notes on advice from the thread: ’s redirect option is valid if you can consolidate to useful pages; ’s removal‑tool reminder is correct about temporariness — treat the Removals tool as a stopgap while you implement the permanent, crawlable noindex. If timeline or behavior looks odd after these steps, gather a few example URLs and their URL Inspection snapshots before escalating (that evidence speeds debugging).

Recommended Answers

All 10 Replies

Member Avatar for Dani

Oh, also ... the pages are noindexed, although I do know the page needs to be seen by Google in order to see there's a noindex directive. I don't want to remove the pages from robots.txt because there's a lot of faceted navigation involved so it would eat up my entire crawl budget. That's why I thought that requesting a temporary removal would do the trick, but it hasn't.

Member Avatar for chrisjames

1.On your computer, go to your Google Account.
2.On the top left navigation panel, click Data & personalization.
3.Under "Activity and timeline," click My Activity.
4.Find the item you want to delete. ...
5.On the item you want to delete, click More .
6.Click Delete.

Member Avatar for Dani

Umm .... that has nothing to do with the question I asked. This is the SEO forum.

Member Avatar for katebonner

Google search displays information collected from websites in the web. The best way to remove information about yourself in Google search results is to contact the website owner who published the information. If they remove it, Google will not be able to find the listing information in the search results.

If the website owner does not remove it, Google will delete certain types of sensitive personal information.

Member Avatar for JeanMilburn

This request works only for pages and images that have already been updated or removed from the web. Use the correct URL in your request.

Member Avatar for Joozed

Ultimately, you can noindex, you can do Robots.txt exclusions, or you can go into Webmaster Tools and try to make the removal, but it's just like indexing in that how and when Google responds to this is really up in the air.

Is the content so important that you can't delete or redirect it entirely? As opposed to messing with indexing and robots.txt, I typically prefer to shoot it to more relevant/higher quality content with a 301.

Member Avatar for Dani

in that how and when Google responds to this is really up in the air.

True, it may take a really long time and it's not actually guaranteed (especially if Google thinks it's a typo or server misconfiguration on your end, they may ignore your directives), but I think you can be confident that if you noindex a page, and Google recrawls it since that meta directive was added, they won't index it. Googlebot also attemps to be a good citizen and they won't crawl pages in robots.txt, although they may still index these pages (especially if they have a lot of backlinks) even though they don't have access to the content on the page.

Member Avatar for rbale

If you really want to get a definitive answer to the issue (provided that you're still experiencing it!), just ask @JohnMu on twitter. He's Google's Senior Search Advocate and usually pretty good at answering problem issues. At the very least you'll get to hear it straight from the horse's mouth!

Hope this helps.

Member Avatar for pradeepmadgaonk

PRO TIP:

It is considered an SEO best practice to ensure all pages redirect to their lower case version because search engines recognize URLs with different cases as being different pages.

Ensure you have followed Google’s Removal Guidelines

Remember that the link removal request that you’ve submitted through Google Search Console is not permanent. After 90 days your content could return to search results pages if you haven’t performed the following actions to block access to your content:

Block URLs with robots.txt
Block URLs by password-protecting your server directories
Block search indexing with meta tags
Opt-out of display on Google+ Local and other Google Properties
With all that in place, you should be good to go!

Member Avatar for HelgaRaymond

Choose “Removal” from the “Index” menu. Select the “Temporary Removals” tab. Hit the “New Request” button (keep it on “Remove this URL only”) and enter the URL of the page you wish to remove entirely from search results and Google's cache. Then click “Submit Request”

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.