0

How does one go about creating a robots.txt file and placing it in a web site? I hear a lot of talk about robots.txt and how important it is for SEO.

6
Contributors
7
Replies
12
Views
6 Years
Discussion Span
Last Post by Ketul_1
0

Well, its a simple text file.

User-agent: *
Allow: /blog
Disallow: /mail

This is the Robots.txt of my website, user-agents determine the bot, i.e., if its google or bing or whatever. Allow tabs tells which part of the domain to be indexed by the SEs. Disallo part orders the bot to not crawl the mail directory.

0

a. with a text editor
b. with an ftp client

[voice=porky]that's all folks[/voice] gotta make a joke sometime

robots.txt is just a text file, notepad will do it, the content is simple there are dozens of short tuts on what it needs to contain to accomplish the litany it is capable of
and being a text file is real easy to reset if something goes awry

0

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

0

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

a: development is done locally and uploaded to the server
no, the op would not require the ftp client first,
or
b: the questions were asked by the op sequentially, and answered in the same sequence

speaks to the skill, or lack thereof, you are bringing to the clients you seek in your signaturego away intellectual amoeba

Edited by almostbob: n/a

-2
                User-agent: *
                Disallow:  /

                           Now, the search engine don't crawl and index your pages.

You add this code to your robots.txt file. Your robots.txt file should be present in root directory of your websites

Votes + Comments
total fail...
This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.