Member Avatar for nabeelarkisdata

is robot.txt files important?
how do we remove it please?

  • 14 Contributors
  • 14 Replies
  • 315 Views
  • 11 Years Discussion Span
  • Latest Post Latest Post by pradeepmadgaonk

Dani AI

Generated

Short answer: yes — robots.txt is important, but not in the way many posters imply. It tells well‑behaved crawlers which URLs to fetch (crawl); it does not by itself reliably remove pages from search results (a blocked URL can still appear without a snippet). Several members here are on the right track: @Fonda and correctly note its role in guiding crawlers, and ’s advice to edit rather than blindly delete is sound. (developers.google.com)

Why pages sometimes stay out of the index: if robots.txt blocks crawling, Google cannot see the page’s HTML (including any meta noindex). That means a meta noindex on a blocked page will never be seen, and the URL can still be listed. Deleting the robots.txt file (causing a 404) generally makes the site appear unrestricted to Google, while server errors (5xx) can temporarily stop crawling; Google may cache robots.txt for up to about 24 hours. Plan edits carefully. (developers.google.com)

Practical checklist (concise, ordered troubleshooting and fix steps):

  • Inspect the live file at the site root (the robots.txt at the host/protocol root controls those URLs). (developers.google.com)
  • Test specific URLs with a modern robots.txt tester (use tools built on Google’s parser to match real behavior). (robotstxt.com)
  • Confirm there are no sitewide meta noindex or X‑Robots‑Tag headers that prevent indexing (these must be visible to crawlers to work). (developers.google.com)
  • If removal of blocks is needed, remove or edit the Disallow lines on the server (editing is safer than deleting in most cases). After changes, request recrawl / resubmit the sitemap via Search Console and monitor coverage — reindexing can take days to weeks. (developers.google.com)

Short cautions and best practices: never use robots.txt as a security measure (it’s public); put the file in the site root and keep it small and well‑formed; prefer targeted edits over wholesale deletion; add sitemap directives so crawlers can find important pages. (developers.google.com)

Summary: edit to unblock the specific paths, ensure pages are accessible (noindex/header rules visible), ask for recrawl, and wait — that sequence fixes the vast majority of “not indexed” problems.

Recommended Answers

All 14 Replies

Member Avatar for Fonda

A well-developed Robots.txt file helps you to get better search engine rankings by providing the important information to the search engine bot.

Using this file only you will tell the search engine crawler which pages of your site should be indexed and which can be ignored.

So it is very important for a site.

Member Avatar for nabeelarkisdata

thanks but my pages are not being indexed. So i want to remove the robot.txt files.

Member Avatar for devisrimari1

hi,
robot text file is used to inform search engine which page have to crawl and which one is not.it makes your site search engine friendly.it can be removed by deleting that file from your root.

Member Avatar for almostbob

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml
Edited by almostbob because: n/a
Member Avatar for peexter

You can block your webpage from Robots.txt and instruction to search engine crawler which folders not crawl. It is helpful.

Member Avatar for micheal_paul

Yes, It is an important on page seo technique..

Member Avatar for Highave

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

Is there necessary to put the last line means point 7 ??

Member Avatar for John Ripple

example of how to see robots.txt :- domain.com/robots.txt

Edited by John Ripple because: n/a
Member Avatar for adityaa

The robot.txt file provides the right way regarding content. It provide the best way when the crawling system going like Google it not accept the spam work.

Member Avatar for kerin1539

Robots.txt file tells the search engine,which Web pages of your site should be crawled and which Web pages should be ignored.

Member Avatar for stunning boy

HI nabeel if your web pages are not being indexed by bots make sure to put the robot.txt on those page which you dont want the spiders to crawl.

Member Avatar for optimumscr12

Robots.txt is good for your website. You can hide your files from crawlers and you can give a path to spiders to crawl your website.
If you want remove your robots.txt file then you can remove it from your googlewebmaster account.

Member Avatar for sam_60

You no need to have any technical experience grasp the power of robots.txt file, just you need to have a source code of your website to use this.

Importance of robots.txt file

  1. It is a text file which guide web robots to crawl which pages on our website.
  2. It also tell web robots which pages not to crawl.
  3. This is very important for site because every site has number of pages, so to crawl every page of your website it will take time and it will get negative ranking on your website.
  4. For example you have lot of pages in which some then are important whuch tell about your website, and some of them are not.
  5. So by generating he code through robot.txt file it will pass the juice to search engines to crawl easily and in less time.
  6. Depending upon the crawling the ranking of the website vaqries.

Thanks.

Member Avatar for pradeepmadgaonk

Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.