7 Years
Discussion Span
Last Post by cguan_77

I thought that a robots.txt file was just to prevent search engines from crawling through folders they shouldn't. I don't think it will increase web traffic. I could be wrong though but that's my understanding of it.


Having a neat robot.txt file, may help a bit with SEO. As Borzoi said, it allows/disallow robots to crawl certain files or folders on your website. If you use the robot.txt file cleverly, you can prevent robots from crawling duplicate content on your site, and this will help with site SEO.

I agree, it will not neccesarily mean more visitors, but will improve your site visibility on search engines if coded correctly, resulting in maybe more visitors.

This question has already been answered. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.