12
Contributors
12
Replies
13
Views
7 Years
Discussion Span
Last Post by optimumscr12
Featured Replies
  • [B]no[/B] you dont want to remove it, you want to edit it to ensure it matches your site structure and requirements[code=apache]User-agent: * Disallow: /cgi-bin/ Disallow: /other_folder/ Disallow: /another_folder/ Disallow: /pics/ Sitemap: http://www.mysite.com/sitemap.xml[/code] Read More

0

A well-developed Robots.txt file helps you to get better search engine rankings by providing the important information to the search engine bot.

Using this file only you will tell the search engine crawler which pages of your site should be indexed and which can be ignored.

So it is very important for a site.

0

hi,
robot text file is used to inform search engine which page have to crawl and which one is not.it makes your site search engine friendly.it can be removed by deleting that file from your root.

1

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

Edited by almostbob: n/a

0

You can block your webpage from Robots.txt and instruction to search engine crawler which folders not crawl. It is helpful.

0

no you dont want to remove it,
you want to edit it to ensure it matches your site structure and requirements

User-agent: *
Disallow: /cgi-bin/
Disallow: /other_folder/
Disallow: /another_folder/
Disallow: /pics/

Sitemap: http://www.mysite.com/sitemap.xml

Is there necessary to put the last line means point 7 ??

0

The robot.txt file provides the right way regarding content. It provide the best way when the crawling system going like Google it not accept the spam work.

0

Robots.txt file tells the search engine,which Web pages of your site should be crawled and which Web pages should be ignored.

0

HI nabeel if your web pages are not being indexed by bots make sure to put the robot.txt on those page which you dont want the spiders to crawl.

0

Robots.txt is good for your website. You can hide your files from crawlers and you can give a path to spiders to crawl your website.
If you want remove your robots.txt file then you can remove it from your googlewebmaster account.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.