User Name Password Register
DaniWeb IT Discussion Community
All
What is DaniWeb IT Discussion Community?
You're currently browsing the Search Engine Optimization section within the Site Management category of DaniWeb, a massive community of 402,006 software developers, web developers, Internet marketers, and tech gurus who are all enthusiastic about making contacts, networking, and learning from each other. In fact, there are 2,387 IT professionals currently interacting right now! Registration is free, only takes a minute and lets you enjoy all of the interactive features of the site.
Please support our Search Engine Optimization advertiser: Affiliate Marketing
Views: 6413 | Replies: 22
Reply
Join Date: Feb 2002
Location: Lawn Guylen, NY
Posts: 10,889
Reputation: cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice 
Rep Power: 32
Solved Threads: 109
Admin
Staff Writer
cscgal's Avatar
cscgal cscgal is online now Online
The Queen of DaniWeb

Using a robots.txt to block links

  #1  
Jun 14th, 2004
I would like to maximize the PR on my forum display and thread display pages, without sacrificing PR to less important pages. (for instance, the newthread.php page)

Now suppose there are 5 links on page A. Normally, page A's PR will be spread out to the 5 pages. But now if I added a robots.txt file which blocked indexing of two of the links. Would page A's PR now be spread less thin among 3 pages? Or would it be spread just as thin, but 2 of the pages would be entitled to a share of PR but just wouldn't use it.

I hope this sorta made a bit of sense.
AddThis Social Bookmark Button
Reply With Quote  
Join Date: Jun 2004
Location: Phoenix Arizona
Posts: 115
Reputation: Arizona Web is an unknown quantity at this point 
Rep Power: 5
Solved Threads: 2
Arizona Web's Avatar
Arizona Web Arizona Web is offline Offline
Junior Poster

Re: Using a robots.txt to block links

  #2  
Jun 14th, 2004
I sent you a PM RE: this topic.
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
Reply With Quote  
Join Date: Feb 2002
Location: Lawn Guylen, NY
Posts: 10,889
Reputation: cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice 
Rep Power: 32
Solved Threads: 109
Admin
Staff Writer
cscgal's Avatar
cscgal cscgal is online now Online
The Queen of DaniWeb

Re: Using a robots.txt to block links

  #3  
Jun 14th, 2004
And I started this thread RE: your PM
Reply With Quote  
Join Date: Jun 2004
Location: Phoenix Arizona
Posts: 115
Reputation: Arizona Web is an unknown quantity at this point 
Rep Power: 5
Solved Threads: 2
Arizona Web's Avatar
Arizona Web Arizona Web is offline Offline
Junior Poster

Re: Using a robots.txt to block links

  #4  
Jun 14th, 2004
Ok I am a knuckelhead. I guess this was a chicken before the egg thing. ;-)
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
Reply With Quote  
Join Date: Feb 2002
Location: Lawn Guylen, NY
Posts: 10,889
Reputation: cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice 
Rep Power: 32
Solved Threads: 109
Admin
Staff Writer
cscgal's Avatar
cscgal cscgal is online now Online
The Queen of DaniWeb

Re: Using a robots.txt to block links

  #5  
Jun 14th, 2004
The robots.txt file I'm currently using is in my root directory (not my forum root) and looks like this:

User-agent: googlebot
Disallow: /techtalkforums/announcement.php
Disallow: /techtalkforums/faq.php
Disallow: /techtalkforums/forumdisplay.php
Disallow: /techtalkforums/login.php
Disallow: /techtalkforums/member.php
Disallow: /techtalkforums/newreply.php
Disallow: /techtalkforums/newthread.php
Disallow: /techtalkforums/online.php
Disallow: /techtalkforums/printthread.php
Disallow: /techtalkforums/search.php
Disallow: /techtalkforums/showthread.php

I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?
Reply With Quote  
Join Date: Jun 2004
Location: Phoenix Arizona
Posts: 115
Reputation: Arizona Web is an unknown quantity at this point 
Rep Power: 5
Solved Threads: 2
Arizona Web's Avatar
Arizona Web Arizona Web is offline Offline
Junior Poster

Re: Using a robots.txt to block links

  #6  
Jun 14th, 2004
without going too far in, it looks decent to me. I am not sure however, if the robots.txt blocks the weakening link pop from all the links. The less links on a page the more potent the links are. A page with tons of links is spreading the pop thin. That would be a good question to ask SEO-Guy.
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
Reply With Quote  
Join Date: Feb 2002
Location: Lawn Guylen, NY
Posts: 10,889
Reputation: cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice 
Rep Power: 32
Solved Threads: 109
Admin
Staff Writer
cscgal's Avatar
cscgal cscgal is online now Online
The Queen of DaniWeb

Re: Using a robots.txt to block links

  #7  
Jun 14th, 2004
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
Reply With Quote  
Join Date: Jun 2004
Location: Phoenix Arizona
Posts: 115
Reputation: Arizona Web is an unknown quantity at this point 
Rep Power: 5
Solved Threads: 2
Arizona Web's Avatar
Arizona Web Arizona Web is offline Offline
Junior Poster

Re: Using a robots.txt to block links

  #8  
Jun 14th, 2004
Originally Posted by cscgal
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
*nods* for sure.
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
Reply With Quote  
Join Date: Jun 2004
Posts: 10
Reputation: AlanM is an unknown quantity at this point 
Rep Power: 5
Solved Threads: 0
AlanM AlanM is offline Offline
Newbie Poster

Re: Using a robots.txt to block links

  #9  
Jun 18th, 2004
I don't see how this will help at all.
Reply With Quote  
Join Date: Feb 2002
Location: Lawn Guylen, NY
Posts: 10,889
Reputation: cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice cscgal is just really nice 
Rep Power: 32
Solved Threads: 109
Admin
Staff Writer
cscgal's Avatar
cscgal cscgal is online now Online
The Queen of DaniWeb

Re: Using a robots.txt to block links

  #10  
Jun 19th, 2004
Google frowns upon multiple pages with the same content. For example, if two different URLs have the exact same content on them, google considers it spamming their search engine. This forum uses Apache's mod_rewrite to rewrite URLs to have a .html extension for search engine purposes. Therefore, the webpage showthread.php?t=100 is the exact same thing as thread100.html - if google spiders see this duplicate contact, they will think that daniweb.com is trying to inflate its page count in google by having multiple URLs with the same content. However, by using robots.txt to block google from spidering the showthread.php pages, google only spiders the pages ending in .html - and therefore doesn't penalize us for duplicate content.
Reply With Quote  
Reply

Only community members can participate in forum threads. You must register or log in to contribute.

DaniWeb Search Engine Optimization Marketplace
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 

Thread Tools Display Modes

Other Threads in the Search Engine Optimization Forum

All times are GMT -4. The time now is 9:32 pm.
Forum system based on vBulletin Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
©2003 - 2008 DaniWeb® LLC