Member Avatar for jennia1686

How does one go about creating a robots.txt file and placing it in a web site? I hear a lot of talk about robots.txt and how important it is for SEO.

  • 6 Contributors
  • 7 Replies
  • 201 Views
  • 3 Years Discussion Span
  • Latest Post Latest Post by Ketul_1

Dani AI

Generated

The replies already in this thread cover the basics: and are right that a robots.txt is just a plain-text file and belongs at the site root, and and demonstrate how directives work. The gaps worth stressing are where people trip up and what to do after the file is created.

Create-and-deploy checklist:

  • Create a plain-text file named robots.txt (UTF-8 without a BOM) and save it at the web host’s document root for the exact host/scheme (each subdomain and each protocol needs its own file).
  • Ensure the webserver serves it with HTTP 200 and Content-Type: text/plain (permissions like 644 are usual).
  • Point crawlers to a sitemap from the file with a Sitemap: line; that helps search engines find pages faster.
  • Verify the result by fetching https://example.com/robots.txt (or using a hosting control panel/FTP to confirm) and by testing it in Google Search Console’s robots.txt tester.

Important cautions and behavior clarifications:

  • robots.txt only controls crawling, not whether a URL can appear in search results. Pages blocked by Disallow can still be indexed if other sites link to them; use a noindex meta tag or an X‑Robots‑Tag header to prevent indexing.
  • Never rely on robots.txt to protect sensitive data: the file is public and can advertise hidden paths.
  • Disallow: / (as shown by ) is useful for staging sites but disastrous on production sites — it prevents well-behaved crawlers from crawling the whole site.
  • Some directives (for example Allow, wildcards, $ end-of-line) are supported by major engines but not by every crawler; for broad compatibility prefer simple User-agent / Disallow rules.

If the file seems ignored, check server redirects, host/scheme mismatch, unexpected 3xx/4xx responses, or a UTF-8 BOM.

Recommended Answers

All 7 Replies

Member Avatar for takeonepicture

Upload robot.txt to your root directory.

Member Avatar for Steven.Matrix

Well, its a simple text file.

User-agent: *
Allow: /blog
Disallow: /mail

This is the Robots.txt of my website, user-agents determine the bot, i.e., if its google or bing or whatever. Allow tabs tells which part of the domain to be indexed by the SEs. Disallo part orders the bot to not crawl the mail directory.

Member Avatar for almostbob

a. with a text editor
b. with an ftp client

[voice=porky]that's all folks[/voice] gotta make a joke sometime

robots.txt is just a text file, notepad will do it, the content is simple there are dozens of short tuts on what it needs to contain to accomplish the litany it is capable of
and being a text file is real easy to reset if something goes awry

Member Avatar for Steven.Matrix

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

Member Avatar for almostbob

Oh lol Bob, I didn't knew that you could edit with a text file.. lol don't you think he needs access to the file first before editing it? Just rearrange the options lol :)

a: development is done locally and uploaded to the server
no, the op would not require the ftp client first,
or
b: the questions were asked by the op sequentially, and answered in the same sequence

speaks to the skill, or lack thereof, you are bringing to the clients you seek in your signaturego away intellectual amoeba

Edited by almostbob because: n/a
Member Avatar for guruparthi 
                User-agent: *
                Disallow:  /

                           Now, the search engine don't crawl and index your pages.

You add this code to your robots.txt file. Your robots.txt file should be present in root directory of your websites

commented: total fail... -3
Member Avatar for Ketul_1

Guru, if he write your code, SE bots never index his page. Please dont misguide him.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.