Thread
:
Using a robots.txt to block links
View Single Post
•
•
Join Date: Feb 2002
Posts: 12,057
Reputation:
Solved Threads: 131
cscgal
Offline
The Queen of DaniWeb
Re: Using a robots.txt to block links
0
#
7
Jun 14th, 2004
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
Dani the Computer Science Gal
Follow my Twitter feed!
twitter.com/DaniWeb
And if you're interested in Internet marketing there is
twitter.com/DaniWebAds
cscgal
View Public Profile
Visit cscgal's homepage!
Find all posts by cscgal