We're a community of 1076K IT Pros here for help, advice, solutions, professional growth and fun. Join us!
1,075,987 Members — Technology Publication meets Social Media
Username:
Password:
Lost login information?
Start New Discussion Reply to this Discussion

Role of Robot.txt in seo

What is robot.txt file and what role it play in the seo. and how can we create it.

11
Contributors
12
Replies
3 Weeks
Discussion Span
1 Year Ago
Last Updated
14
Views
ilmkidunya
Light Poster
43 posts since Dec 2011
Reputation Points: 7
Solved Threads: 0
Skill Endorsements: 0

The file lets you request robots not to index certain files.

With relation to SEO, it helps you control which pages get indexed. So you can have pages that have no SEObenefit excluded from indexing. Also, restricting the number of pages that can be indexed increases the chance that your valuable pages will get indexed more frequently.

There are 1,000s of articles on the web to explain how to write one.

Tiggerito
Junior Poster in Training
59 posts since Jul 2010
Reputation Points: 10
Solved Threads: 5
Skill Endorsements: 0

Robots.txt file is used to tell the search engines which pages to be indexed by and which pages not to index. Robots.txt is file is used to block the admin pages in the site

claydcousta
Junior Poster in Training
64 posts since Jun 2011
Reputation Points: 7
Solved Threads: 0
Skill Endorsements: 0

Thanks for your value able information, just one more question,

how can i create the text file is there any software for that or manually creates the file thanks

ilmkidunya
Light Poster
43 posts since Dec 2011
Reputation Points: 7
Solved Threads: 0
Skill Endorsements: 0

Robot.txt helps you to cover or not to show any files to search engine. It hides it from search engine.

jhonden
Newbie Poster
Banned
17 posts since Sep 2011
Reputation Points: -5
Solved Threads: 4
Skill Endorsements: 0

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. Robots.txt file is to improve site indexation by telling search engine crawler to only index your content pages and to ignore other pages (i.e. monthly archives, categories folders or your admin files) that you do not want them to appear on the search index.

Braylen
Newbie Poster
Banned
22 posts since Dec 2011
Reputation Points: 1
Solved Threads: 1
Skill Endorsements: 0

Robots.txt file is used to tell the search engines which pages to be indexed by and not to index. Means this is the file where we can allow to searching engine what you have to follow or what not.

Create Robot txt file:

jonywags
Newbie Poster
Banned
18 posts since Dec 2011
Reputation Points: -6
Solved Threads: 1
Skill Endorsements: 0

Thank you jonywags is solves my problem....

ilmkidunya
Light Poster
43 posts since Dec 2011
Reputation Points: 7
Solved Threads: 0
Skill Endorsements: 0

Hi friends, robot.txt file basically used to tell search engine which page crawl and which are not.................Thanks,

vinodkunwar
Newbie Poster
Banned
23 posts since Dec 2011
Reputation Points: -2
Solved Threads: 0
Skill Endorsements: 0

What is robot.txt file and what role it play in the seo. and how can we create it.

They retrieved link pages to your website, they are the ones telling the google spiders what to visit and what are not.

trishia
Light Poster
25 posts since Dec 2011
Reputation Points: 10
Solved Threads: 0
Skill Endorsements: 0

it instruct search engines robot to crawl your website

dailyearner
Junior Poster
142 posts since Apr 2009
Reputation Points: -6
Solved Threads: 5
Skill Endorsements: 0

Robots.txt file is used to block any your of a websites which you don't want to get index by Search Engines.

pankajrocks
Newbie Poster
22 posts since Dec 2011
Reputation Points: -8
Solved Threads: 3
Skill Endorsements: 0

robots are performing the link building.mostly doing the off site

ADRIANEra
Junior Poster in Training
55 posts since Nov 2011
Reputation Points: -5
Solved Threads: 5
Skill Endorsements: 0

This article has been dead for over three months: Start a new discussion instead

Post: Markdown Syntax: Formatting Help
 
You
View similar articles that have also been tagged:
 
© 2013 DaniWeb® LLC
Page rendered in 0.0977 seconds using 2.68MB