Is robot.txt files important?

[nabeelarkisdata]

is robot.txt files important?
how do we remove it please?

[Fonda]

A well-developed Robots.txt file helps you to get better search engine rankings by providing the important information to the search engine bot.

Using this file only you will tell the search engine crawler which pages of your site should be indexed and which can be ignored.

So it is very important for a site.

[nabeelarkisdata]

thanks but my pages are not being indexed. So i want to remove the robot.txt files.

[devisrimari1]

hi,
robot text file is used to inform search engine which page have to crawl and which one is not.it makes your site search engine friendly.it can be removed by deleting that file from your root.

[almostbob]

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

[peexter]

You can block your webpage from Robots.txt and instruction to search engine crawler which folders not crawl. It is helpful.

[micheal_paul]

Yes, It is an important on page seo technique..

[Highave]

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

Is there necessary to put the last line means point 7 ??

[John Ripple]

example of how to see robots.txt :- domain.com/robots.txt
http://vibgyortel.in/robots.txt

[adityaa]

The robot.txt file provides the right way regarding content. It provide the best way when the crawling system going like Google it not accept the spam work.

[kerin1539]

Robots.txt file tells the search engine,which Web pages of your site should be crawled and which Web pages should be ignored.

[stunning boy]

HI nabeel if your web pages are not being indexed by bots make sure to put the robot.txt on those page which you dont want the spiders to crawl.

[optimumscr12]

Robots.txt is good for your website. You can hide your files from crawlers and you can give a path to spiders to crawl your website.
If you want remove your robots.txt file then you can remove it from your googlewebmaster account.

[sam_60]

You no need to have any technical experience grasp the power of robots.txt file, just you need to have a source code of your website to use this.

Importance of robots.txt file

1. It is a text file which guide web robots to crawl which pages on our website.
2. It also tell web robots which pages not to crawl.
3. This is very important for site because every site has number of pages, so to crawl every page of your website it will take time and it will get negative ranking on your website.
4. For example you have lot of pages in which some then are important whuch tell about your website, and some of them are not.
5. So by generating he code through robot.txt file it will pass the juice to search engines to crawl easily and in less time.
6. Depending upon the crawling the ranking of the website vaqries.

Thanks.

[pradeepmadgaonk]

Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.