View Single Post
Jun 14th, 2004
0

Re: Using a robots.txt to block links

Quote originally posted by cscgal ...
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
*nods* for sure.
Reputation Points: 16
Solved Threads: 2
Junior Poster
Arizona Web is offline Offline
118 posts
since Jun 2004