 |  |  |  |  |  |  |  |
Using a robots.txt to block links
Please support our Search Engine Optimization advertiser: Search Engine Optimization Services
 |
•
•
Join Date: Feb 2002
Posts: 12,019
Reputation: 
Solved Threads: 129
I would like to maximize the PR on my forum display and thread display pages, without sacrificing PR to less important pages. (for instance, the newthread.php page)
Now suppose there are 5 links on page A. Normally, page A's PR will be spread out to the 5 pages. But now if I added a robots.txt file which blocked indexing of two of the links. Would page A's PR now be spread less thin among 3 pages? Or would it be spread just as thin, but 2 of the pages would be entitled to a share of PR but just wouldn't use it.
I hope this sorta made a bit of sense.
Now suppose there are 5 links on page A. Normally, page A's PR will be spread out to the 5 pages. But now if I added a robots.txt file which blocked indexing of two of the links. Would page A's PR now be spread less thin among 3 pages? Or would it be spread just as thin, but 2 of the pages would be entitled to a share of PR but just wouldn't use it.
I hope this sorta made a bit of sense.
I sent you a PM RE: this topic. 
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
•
•
Join Date: Feb 2002
Posts: 12,019
Reputation:
Solved Threads: 129
And I started this thread RE: your PM
Ok I am a knuckelhead. I guess this was a chicken before the egg thing. ;-)
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
•
•
Join Date: Feb 2002
Posts: 12,019
Reputation:
Solved Threads: 129
The robots.txt file I'm currently using is in my root directory (not my forum root) and looks like this:
I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?
User-agent: googlebot Disallow: /techtalkforums/announcement.php Disallow: /techtalkforums/faq.php Disallow: /techtalkforums/forumdisplay.php Disallow: /techtalkforums/login.php Disallow: /techtalkforums/member.php Disallow: /techtalkforums/newreply.php Disallow: /techtalkforums/newthread.php Disallow: /techtalkforums/online.php Disallow: /techtalkforums/printthread.php Disallow: /techtalkforums/search.php Disallow: /techtalkforums/showthread.php
I am disallowing access to showthread.php and forumdisplay.php because I would rather Google only spider the .html mod_rewrite versions of the forums and threads, and therefore not get duplicate content. Was this done correctly? Am I excluding the correct things?
without going too far in, it looks decent to me. I am not sure however, if the robots.txt blocks the weakening link pop from all the links. The less links on a page the more potent the links are. A page with tons of links is spreading the pop thin. That would be a good question to ask SEO-Guy.
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
•
•
Join Date: Feb 2002
Posts: 12,019
Reputation:
Solved Threads: 129
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
•
•
•
•
Originally Posted by cscgal
It would be very nice if the robots.txt would block the weakening spread of PR. However, even if it doesn't do this, it would still be valuable because it would eliminate spidering duplicate content (i.e. showthread.php?t=10 and thread10.html)
Need a website designer? arizona web design : phoenix web design : MCP Media intelligent web design and web development solutions. MCP Media is owned and operated by Chris Hooley - who happens to be a real nerd... on purpose :-)
•
•
Join Date: Feb 2002
Posts: 12,019
Reputation:
Solved Threads: 129
Google frowns upon multiple pages with the same content. For example, if two different URLs have the exact same content on them, google considers it spamming their search engine. This forum uses Apache's mod_rewrite to rewrite URLs to have a .html extension for search engine purposes. Therefore, the webpage showthread.php?t=100 is the exact same thing as thread100.html - if google spiders see this duplicate contact, they will think that daniweb.com is trying to inflate its page count in google by having multiple URLs with the same content. However, by using robots.txt to block google from spidering the showthread.php pages, google only spiders the pages ending in .html - and therefore doesn't penalize us for duplicate content.
|
Other Threads in the Search Engine Optimization Forum
- Previous Thread: Help.
- Next Thread: Search Engines into LSI
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Related Forum Features
Latest Search Engine Optimization Posts
- Today's Posts
- Unanswered Threads
advertising affordable airplane alexa analytics apple archive audioanalytics backlinks bing biometrics blogs bomb business codesearch company consumer design development digital directory engine england enterprisesearch eric_schmidt facebook flashwebsiteoptimizationtips free gaudi google googleanalytics googlelongersearchbox googlesearch googlesitesearch googletrends hacking imagerecognition imagesearch indexedpage india internet internetmarketingcompanyindia jcpenney keyword keywords kumar leweb link marketing microsoft news no-follow offshoreoutsourcingcompany opensource optimization optimizingforbing pagerank posts powerset ranking refresh revenue rss search searchengine searchengineland searchengineoptimization searchengines seinfeld semanticweb seo seocompany seooptimizationindia seoservices serps services sets sex socialnetworking space spam stalking testing tools tweetnews twitter usenet video videosearch viralmarketing wave web webdesignindia website wolfram wolframalpha wordpress yahoo yahoo! youtube