Re: Lots of new members but no one posts Community Center Meta DaniWeb by Dani … members have their profiles at /profiles/ which is blocked via robots.txt, so crawlers never even go there. (In fact, we… Re: Anti-Hotlinking Script for WP on Apache (.htaccess) - Linkspam Prevention Programming Web Development by Dani CDNs such as Cloudflare offer hotlink protection via a toggle switch you could enable. But, yeah, the primary benefit of hotlink protection is that a website doesn't upload images to *your* site, and then include them in their own site without having to pay for the bandwidth of hosting the images. Just make sure that if you turn on hotlink … Re: Do backlinks to disallowed URLs help the domain overall? Digital Media Digital Marketing Search Engine Strategies by ashleydent4u … that Google may still consider backlinks to URLs disallowed in robots.txt, but it doesn’t mean that the link juice… Re: Do backlinks to disallowed URLs help the domain overall? Digital Media Digital Marketing Search Engine Strategies by MasoodDidThat The sole purpose of disavowing a link is to stop Google from indexing the backlink so it does not affect your ranking. You can upload the file of your disallowed backlinks to GSC (https://search.google.com/search-console/disavow-links). This practice helps protect your site from potential penalties associated with low-quality or spammy links. So… Re: Google Search Console -> Page indexing Digital Media Digital Marketing Search Engine Strategies by Olu_610 Errors in robots.txt file settings and sitemap failures frequently trigger indexing issues. Verify if the robots.txt permits Googlebot to explore all pages and make sure your sitemap is fully updated with proper submission to Search Console. Re: Google Search Console -> Page indexing Digital Media Digital Marketing Search Engine Strategies by ashleydent4u … like you're on the right track already, and your robots.txt file actually looks okay at first glance. However, if… Re: Page Crawling Issue Digital Media Digital Marketing by ashleydent4u … be due to your crawl settings or restrictions. Check your robots.txt file and meta tags to ensure SEMrush isn’t… Re: Page Crawling Issue Digital Media Digital Marketing by jonathannweyer If SEMRush is only crawling a few pages, it might be due to the crawl settings or limits in your SEMRush account. Check if there are any crawl restrictions set up and make sure your robots.txt file isn’t blocking any important pages. robots.txt and sitemap.xml Digital Media Digital Marketing Search Engine Strategies by cmps … like to know what happens if I wrote in the robots.txt: User-agent: * Disallow: / Sitemap: http://www.example.com/…In this case, as I understand what robots.txt and sitemap.xml do; First, robots.txt disallow Search Engine (Let's talk…to know, what happens when a page has the meta robots set to noindex, but at the same time it appears… Re: robots.txt and 302 redirects Digital Media Digital Marketing Search Engine Strategies by joeyoungblood blocking the URL in the robots.txt doesn't do much good these days. Google will … want and rank it for what they want. noindex meta robots tag is far more useful. you say it worked, but…. in late June google posted about using robots.txt vs noindex and stated that robots.txt was nolonger their endorsed method. [url… Robots.txt explanation with Video. Digital Media Digital Marketing Search Engine Strategies by stephen186 Those who are still confused about robots.txt. Here is a link from mattcutts. [url]http://www.mattcutts.com/blog/robots-txt-remove-url/[/url] Re: Robots.txt explanation with Video. Digital Media Digital Marketing Search Engine Strategies by mahgobb [QUOTE=stephen186;1010704]Those who are still confused about robots.txt. Here is a link from mattcutts. [url]http://www.mattcutts.com/blog/robots-txt-remove-url/[/url][/QUOTE] Thanks for the sharing of this link, indeed, it clear me some doubts! Re: Robots at war Community Center Geeks' Lounge by cwarn23 I wonder how long until they invent replicators. Replicators would be able to wipe out the robots with no problem. Or what about hackers hacking into a robot network. A hacker could send a Trojan to all robots in the United States and give those robots the command to destroy everything in sight. G' the future hackers have a lot more to play with... Re: Robots at war Community Center Geeks' Lounge by vegaseat … of these questions. It also presents the idea that the robots get more and more sophisticated with time and start to… some of the basic rules. BTW, they use nano robots to destroy uncooperative robots. Kind of goofy, since Will Smith just blows… Robots Help Ill Children Achieve Mobility Community Center Geeks' Lounge by Niki_Fears … progression. Of course this is not the first time that robots have been used for medical purposes. They have been successfully… success to gain the project, and the positive image of robots, even more exposure. robots.txt for subdomains Digital Media Digital Marketing Search Engine Strategies by webmaster@123 I have created a sub domain and put robots file As follows User-agent: * DisAllow: / Will it affect my main domain? I have robots file in my main domain as follows User-agent: * Allow: / Robots.txt Checker Programming Web Development by hacxx If a site has the file "robots.txt", it displays all the directories and pages on the website that they wish to keep anonymous from the search engine spiders. This may lead to hidden pages. With Robots.txt Checker you can access the site directly or check if it is indexed in google Re: Robots.txt file Digital Media Digital Marketing Search Engine Strategies by almostbob [code=robots.txt]User-agent: * Disallow: /temp/ Allow: /temp/abc.html[/code]… up in someone's web server log as a Referer. /robots.txt is not intended for access control it's a… Re: Robots at war Community Center Geeks' Lounge by GrimJack Heh,heh - there are already self-repairing surfaces. The robots that come for you might not look like what you …-doll[/URL], [URL="http://io9.com/5280536/new-tiny-robots-can-crawl-through-your-veins"]living in your body… Re: Robots Help Ill Children Achieve Mobility Community Center Geeks' Lounge by Niki_Fears Lol, not at all. Robots have a variety of great uses, including therapies such as this and, if we can ever get past our "jobs" capitalist obsessed mind set, they could change the way we do a lot of things in the future. Re: robots.txt and 302 redirects Digital Media Digital Marketing Search Engine Strategies by Dani It's been a couple of days, and Google Webmaster Tools is now showing me that page1.html is not being crawled due to being blocked in my robots.txt file, even though it is only page2.html that is actually listed in robots.txt. This is the desired effect, in my case. Re: robots.txt for subdomains Digital Media Digital Marketing Search Engine Strategies by jorik This will affect your main domain only if it is accessible through your main domain url also. e.g. yourmaindomain .com/robots.txt Re: robots.txt disallowing certain query stings... Digital Media UI / UX Design by Ali_42 Read the Correct Guidlines about Robots txt http://tools.seobook.com/robots-txt/ Robots.txt Digital Media Digital Marketing Search Engine Strategies by SuperTodd I dont want any of the pages to be blocked from the Search Engines. Do i need to place a blank robots.txt file please advise thanks Re: Robots.txt Digital Media Digital Marketing Search Engine Strategies by stymiee No. Only place a blank robots.txt if you don't want 404 errors in your logs/stat software. Otherwise there is no need for one. Re: Robots.txt Digital Media Digital Marketing Search Engine Strategies by Harry smith You want each and every page of your site should be index by every spider you can use a simple two line code in your robots.txt User-agent: * Disallow: Here * indicates all spiders are allowed to crawl and the disallow section is placed empty, this indicates every portion of the site is free for them to crawl. Re: Robots.txt Digital Media Digital Marketing Search Engine Strategies by Dani ... or you can just not have a robots.txt file to achieve the same effect. Re: Robots.txt Digital Media Digital Marketing Search Engine Strategies by BamaStangGuy A robots.txt file is for blocking content from search engines. This is helpful if you do not want search engines to "waste" their time attempting to index such things that: [list]Won't give you any traffic from SERPs such[/list] [list]Only accessible by members[/list] Robots Result Digital Media Digital Marketing Search Engine Strategies by harininet Hi, I have a quick question! I have implemeted robots.txt to disallow some of my pages! How long it will take to do this? whether i can get all my pages OUT OF INDEX all at a time or it will happen pages by pages. I'm confused Ideas are welcome Re: Robots Result Digital Media Digital Marketing Search Engine Strategies by stymiee It depends on your site. Sites that are crawled more frequently will see results faster then sites crawled less frequently. If you want pages removed from the SERPs quickly robots.txt is not your best option. Use a removal request instead.