Member Avatar for Dani

I have page1.html that is being 302 redirected (temporary redirect) to page2.html
page2.html is disallowed in my robots.txt file

Under normal circumstances, when googlebot encounters a 301 redirect from page1.html to page2.html, it will index page2.html, and when googlebot encounters a 302 redirect from page1.html to page2.html, it will index page1.html

Since, theoretically, the url of page1.html is what would be indexed, would it still be indexed considering page2.html is blocked?

  • 4 Contributors
  • 5 Replies
  • 1K Views
  • 4 Months Discussion Span
  • Latest Post Latest Post by joeyoungblood

Dani AI

Generated

Short answer: that Search Console result is expected. When Googlebot fetches page1.html and sees a 302 to page2.html it will follow the redirect and then check the destination. If the final URL is disallowed by robots.txt, Google cannot fetch the target content and will often report the original URL as “blocked by robots.txt” in Search Console (the crawler checks robots.txt for the redirect target). Google follows redirects (301 is treated as a stronger canonical signal than 302) and will try to consider the final target for indexing only when it can fetch it. (developers.google.com)

Important distinction and practical fixes: robots.txt blocks crawling, not indexing — a disallowed URL can still appear in search results without a snippet. To prevent a URL from appearing in Search, use a crawlable noindex (meta tag or X-Robots-Tag) or require authentication; do not disallow the page in robots.txt if relying on noindex, because Googlebot must be allowed to fetch the page to see the directive. If the redirect is meant to be permanent and the new page should be canonical, use a 301 and allow the target to be crawled. If the intent is to keep the target hidden, remove the robots disallow and add a noindex (or use auth / 404/410). (developers.google.com)

Quick troubleshooting checklist: run URL Inspection and the robots.txt tester in Search Console; trace the redirect chain with curl -I or a redirect-trace tool to confirm each hop and the HTTP status; decide whether the move is temporary (302) or permanent (301) and whether the target must be crawlable for a noindex to work; then request a recrawl via Search Console. (developers.google.com)

Notes: this explains ’s GWT finding; ’s instinct that a redirect influences crawling was on track, and ’s warning about robots.txt vs noindex is consistent with current Google guidance.

Recommended Answers

All 5 Replies

Member Avatar for canadafred

I would think that because you redirect page1 to page2 the search engine will include it in a crawl despite a robot.txt instruction to do otherwise.

Member Avatar for Dani

Would it just crawl (b/c when it first finds page1.html, it is a valid url for it), or would it actually index the contents of page2.html, despite a robots.txt file to disallow crawling or indexing of page2.html?

Member Avatar for Dani

It's been a couple of days, and Google Webmaster Tools is now showing me that page1.html is not being crawled due to being blocked in my robots.txt file, even though it is only page2.html that is actually listed in robots.txt.

This is the desired effect, in my case.

Member Avatar for sugeshg

If you had blocked your page2.html in robots.txt. The search engines bots won't crawl that page even though you had (302 - 'Found' or 'Moved Temporarily') redirected the page1.html to page2.html.

User-agent: *
Disallow: /page2.html

Confirm that you had verified your domain name in webmaster central.

http://www.google.com/webmasters/

Resubmit your sitemap.xml having page1.html in Google Webmaster Tools and Bing Webmaster Center. The SE bots will crawl the URLS given in sitemap.xml and update their index accordingly.

Member Avatar for joeyoungblood

blocking the URL in the robots.txt doesn't do much good these days. Google will still index the URL and give it whatever title they want and rank it for what they want. noindex meta robots tag is far more useful.

you say it worked, but I would keep an eye on it. in late June google posted about using robots.txt vs noindex and stated that robots.txt was nolonger their endorsed method.

they have since clarified that they WILL index the URL but not the page or it's content. that means you can easily run into duplicate/thin content issues by blocking URLs that might get shared out on the web.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.