I have a quick question! I have implemeted robots.txt to disallow some of my pages! How long it will take to do this? whether i can get all my pages OUT OF INDEX all at a time or it will happen pages by pages. I'm confused

Ideas are welcome

Recommended Answers

All 4 Replies

It depends on your site. Sites that are crawled more frequently will see results faster then sites crawled less frequently. If you want pages removed from the SERPs quickly robots.txt is not your best option. Use a removal request instead.

when the seach engine crawls your site the first file it hits is robots.txt and if it sees to disallow any files it won't index them which means they should go off there index!

when the bot crawls thats the time when it'll see what is allowed and whats not allowed and make sure u use opening slash / and closing slash /


something like that when not allowing the bot to crawl with the right syntax =)

Yes, I agree with webmasterran. I too use robot,txt for my sites to say the search engines not to crawl my page.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.