 |  |  |  |  |  |  |  |
Using a robots.txt to block links
Please support our Search Engine Optimization advertiser: Search Engine Optimization Services
 |
may i aks how you changed it to thread6988.html instead of showthread?
Xtremecomputing #1 For Hardware News And Reviews
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
http://www.xtremecomputing.co.uk/for...ilies/cool.gif [ XP 3000+ | ATi Radeon Pro 9800 http://www.xtremecomputing.co.uk/for...lies/wooha.gif | 512mb DDr | 4x Pioneer DVD-RW | 16x DVD | 120gb Maxtor Hard Disk | 17" Flatscreen | Sblive Platinum | Creative Inspire 5.1 ] http://www.xtremecomputing.co.uk/for...milies/eek.gif
•
•
•
•
may i aks how you changed it to thread6988.html instead of showthread?
On this server, the page thread6988.htm does not physically exist. Instead, the web server monitors incoming url requests and looks for the word thread in that request..... if so, it grabs the numbers from that and passes it along to showthread.php easy enough.
Hope this helps.
•
•
Join Date: Feb 2005
Posts: 252
Reputation: 
Solved Threads: 1
Posting Whiz in Training
Would it make sense to use the robots no follow tag in your particular case?
www.markdavidny.com - just one of my sites I'm trying to boost traffic for ;-)
Yes, i need the robots.txt from daniweb for my vbulletin forum: http://www.schachfeld.de/
Where can I find the robots.txt?
Where can I find the robots.txt?
•
•
Join Date: Feb 2002
Posts: 12,024
Reputation: 
Solved Threads: 126
Post #5 shows the robots.txt file that I used to use. I no longer use a robots.txt file.
•
•
•
•
Originally Posted by cscgal
Post #5 shows the robots.txt file that I used to use. I no longer use a robots.txt file.
My robots.txt file: http://www.schachfeld.de/robots.txt
We had a problem where pages that had a no-crawl code at the root directory still were being crawled (they were PDFs that had valuable IP in them).
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
We discovered that the bots were getting in through links on other pages of ours (the PDFs are "samples" of products that we use as marketing tools), so we put "no follow" codes -- <meta name="robots" content="index,nofollow" /> -- on those pages.
This let's the spider index the page but not follow the links on the page.
But if someone includes a link to the non-HTML thread in a page that you don't control, do you think it will bypass your html rewrite?
|
Other Threads in the Search Engine Optimization Forum
- Previous Thread: Help.
- Next Thread: Search Engines into LSI
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Related Forum Features
Latest Search Engine Optimization Posts
- Today's Posts
- Unanswered Threads
adsense advertising agency analytics apple backlink backlinks bing blog blogs british bug business cheap community consumersearch design development earth email engine enterprisesearch europe facebook failure feeds filters flash flashoptimizationtips free google googlebot inboundlinks india internetmarketing jobsearch keywords krugle laws link live livesearch marissamayer marketing men microsoft motrin multi-function news nexidia offshoredevelopmentcompanyindia pagerank photo plug-ins professional ranking ratings repository research seach search searchengine searchengineoptimization searchwithoutkeywords semanticweb seo seoemails seooptimization seooptimizationindia seoservices sergey_brinn silverlight site skills socialnetworking strategies sullivan tagging targetspot tool travel tweet twitter universe video videos videosearch viralmarketing vista web webdevelopmentcompanyindia webmaster websearch website wolfram wolframalpha women yahoo yahoo! youtube