Thread: Using a robots.txt to block links
View Single Post
•
•
Join Date: Feb 2002
Posts: 12,024
Reputation: cscgal is a glorious beacon of light cscgal is a glorious beacon of light cscgal is a glorious beacon of light cscgal is a glorious beacon of light cscgal is a glorious beacon of light cscgal is a glorious beacon of light 
Solved Threads: 126
Administrator
Staff Writer
cscgal's Avatar
cscgal cscgal is offline Offline
The Queen of DaniWeb

Re: Using a robots.txt to block links

 
0
  #7
Jun 14th, 2004
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
Dani the Computer Science Gal
Follow my Twitter feed! twitter.com/daniweb
Reply With Quote