 |  |  |  |  |  |  |  |
Using a robots.txt to block links
Please support our Search Engine Optimization advertiser: Get a Free SEO Analysis!
 |
may i aks how you changed it to thread6988.html instead of showthread?
Xtremecomputing #1 For Hardware News And Reviews
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
•
•
•
•
may i aks how you changed it to thread6988.html instead of showthread?
On this server, the page thread6988.htm does not physically exist. Instead, the web server monitors incoming url requests and looks for the word thread in that request..... if so, it grabs the numbers from that and passes it along to showthread.php easy enough.
Hope this helps.
•
•
Join Date: Feb 2005
Posts: 252
Reputation: 
Solved Threads: 1
Posting Whiz in Training
Would it make sense to use the robots no follow tag in your particular case?
www.markdavidny.com - just one of my sites I'm trying to boost traffic for ;-)
Yes, i need the robots.txt from daniweb for my vbulletin forum: http://www.schachfeld.de/
Where can I find the robots.txt?
Where can I find the robots.txt?
•
•
Join Date: Feb 2002
Posts: 12,036
Reputation: 
Solved Threads: 130
Post #5 shows the robots.txt file that I used to use. I no longer use a robots.txt file.
•
•
•
•
Originally Posted by cscgal
Post #5 shows the robots.txt file that I used to use. I no longer use a robots.txt file.
My robots.txt file: http://www.schachfeld.de/robots.txt
We had a problem where pages that had a no-crawl code at the root directory still were being crawled (they were PDFs that had valuable IP in them).
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
|
Other Threads in the Search Engine Optimization Forum
- Previous Thread: Help.
- Next Thread: Search Engines into LSI
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Related Forum Features
Latest Search Engine Optimization Posts
- Today's Posts
- Unanswered Threads
advertising affordable airplane alexa analytics apple archive audioanalytics backlinks bing biometrics block blog blogs business codesearch consumer design development digital directory engine england enterprisesearch eric_schmidt estate facebook flashwebsiteoptimizationtips gaudi google googleanalytics googlelongersearchbox googlesearch googlesitesearch googletrends imagerecognition imagesearch indexedpage india internet internetmarketingcompanyindia jcpenney keyword keywords kumar leweb marketing microsoft news no-follow opensource optimization optimizingforbing organic pagerank plug-ins posts powerset ppc propertly ranking refresh revenue search searchengine searchengineland searchengineoptimization searchengines seinfeld semanticweb seo seocompany seooptimizationindia seoservices serps services sets sex socialnetworking space spam stalking testing tools tweetnews twitter usenet video videosearch viralmarketing wave web webdesignindia website wolfram wolframalpha wordpress yahoo yahoo! youtube
