What is robot.txt? Is it necessary to have robot.txt for a webpage? Does it effect SEO at all?


Recommended Answers

All 2 Replies

There is quite a bit of information out there regarding the use of the robot.txt file. If you as the web site owner, want to give specific instructions to web spiders/robots, you would place a text file called robots.txt in the root of the web site directory. This text file should include instructions in a specific format that instruct the robot on whether to allow or deny it access to folders and/or files.

So, if you have a certain directory or files that you do not want to be indexed by a search engine, you should definately include this file.

If you get an account with Google Webmasters, you can use their online tool to see the effects of a robots.txt file on your site.

You can allow or disallow search engines to crawl your site. Robot.txt is helpful for crawling a site by SE spiders.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, learning, and sharing knowledge.