| | |
Using a robots.txt to block links
Please support our Search Engine Optimization advertiser: Get a Free SEO Analysis!
![]() |
•
•
Join Date: Jun 2004
Posts: 5
Reputation:
Solved Threads: 0
may i aks how you changed it to thread6988.html instead of showthread?
Xtremecomputing #1 For Hardware News And Reviews
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
•
•
Join Date: Jul 2004
Posts: 32
Reputation:
Solved Threads: 1
•
•
•
•
may i aks how you changed it to thread6988.html instead of showthread?
On this server, the page thread6988.htm does not physically exist. Instead, the web server monitors incoming url requests and looks for the word thread in that request..... if so, it grabs the numbers from that and passes it along to showthread.php easy enough.
Hope this helps.
•
•
Join Date: Feb 2005
Posts: 252
Reputation:
Solved Threads: 1
Would it make sense to use the robots no follow tag in your particular case?
www.markdavidny.com - just one of my sites I'm trying to boost traffic for ;-)
•
•
Join Date: Jun 2005
Posts: 9
Reputation:
Solved Threads: 0
Yes, i need the robots.txt from daniweb for my vbulletin forum: http://www.schachfeld.de/
Where can I find the robots.txt?
Where can I find the robots.txt?
•
•
Join Date: Jun 2005
Posts: 9
Reputation:
Solved Threads: 0
•
•
•
•
Originally Posted by cscgal
Post #5 shows the robots.txt file that I used to use. I no longer use a robots.txt file.
My robots.txt file: http://www.schachfeld.de/robots.txt
•
•
Join Date: Jul 2005
Posts: 15
Reputation:
Solved Threads: 0
We had a problem where pages that had a no-crawl code at the root directory still were being crawled (they were PDFs that had valuable IP in them).
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
![]() |
Other Threads in the Search Engine Optimization Forum
- Previous Thread: Help.
- Next Thread: Search Engines into LSI
| Thread Tools | Search this Thread |
advertising affordable airplane alexa analytics apple archive audioanalytics backlinks bing biometrics block blog blogs business codesearch consumer design development digital directory engine england enterprisesearch eric_schmidt estate facebook flashwebsiteoptimizationtips gaudi google googleanalytics googlelongersearchbox googlesearch googlesitesearch googletrends imagerecognition imagesearch indexedpage india internet internetmarketingcompanyindia jcpenney keyword keywords kumar leweb marketing microsoft news no-follow opensource optimization optimizingforbing organic pagerank plug-ins posts powerset ppc propertly ranking refresh revenue search searchengine searchengineland searchengineoptimization searchengines seinfeld semanticweb seo seocompany seooptimizationindia seoservices serps services sets sex socialnetworking space spam stalking testing tools tweetnews twitter usenet video videosearch viralmarketing wave web webdesignindia website wolfram wolframalpha wordpress yahoo yahoo! youtube







