Is robot.txt files important?

[nabeelarkisdata]

is robot.txt files important?
how do we remove it please?

[Fonda]

A well-developed Robots.txt file helps you to get better search engine rankings by providing the important information to the search engine bot.

Using this file only you will tell the search engine crawler which pages of your site should be indexed and which can be ignored.

So it is very important for a site.

[nabeelarkisdata]

thanks but my pages are not being indexed. So i want to remove the robot.txt files.

[devisrimari1]

hi,
robot text file is used to inform search engine which page have to crawl and which one is not.it makes your site search engine friendly.it can be removed by deleting that file from your root.

[almostbob]

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

[peexter]

You can block your webpage from Robots.txt and instruction to search engine crawler which folders not crawl. It is helpful.

[micheal_paul]

Yes, It is an important on page seo technique..

[Highave]

no you dont want to remove it, you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

Is there necessary to put the last line means point 7 ??

[John Ripple]

example of how to see robots.txt :- domain.com/robots.txt
http://vibgyortel.in/robots.txt

[adityaa]

The robot.txt file provides the right way regarding content. It provide the best way when the crawling system going like Google it not accept the spam work.

[kerin1539]

Robots.txt file tells the search engine,which Web pages of your site should be crawled and which Web pages should be ignored.

[stunning boy]

HI nabeel if your web pages are not being indexed by bots make sure to put the robot.txt on those page which you dont want the spiders to crawl.

[optimumscr12]

Robots.txt is good for your website. You can hide your files from crawlers and you can give a path to spiders to crawl your website.
If you want remove your robots.txt file then you can remove it from your googlewebmaster account.