954,360 Members — Technology Publication meets Social Media
Username:
Password:
Lost login information?
Have something to say? Contribute New Article Reply to this Article

robots.txt file syntax

hi,
i have a robots.txt file with this content:
User-agent:*
Allow: /
Allow: /sitemap
Disallow: /gerente

is it written in right order?

fumnimda
Junior Poster in Training
60 posts since Oct 2009
Reputation Points: 10
Solved Threads: 1
 

Hi,

I think these 2 lines would be not useful Allow: / Allow: /sitemap for any purpose since they are allowed by default. Also, might I suggest the contents on robotstxt.org/orig.html#examples for you to read.

all the best,

AirForceOne
Posting Pro in Training
457 posts since Jun 2009
Reputation Points: 19
Solved Threads: 15
 

My robots.txt file looks like this:

Sitemap: http://www.daniweb.com/sitemap.xml

User-agent: Googlebot
Disallow: ...........


It sits in the root directory. I think that Allow is explicit.

cscgal
The Queen of DaniWeb
Administrator
19,421 posts since Feb 2002
Reputation Points: 1,474
Solved Threads: 229
 

This question has already been solved

Post: Markdown Syntax: Formatting Help
You