Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by Dani … Screaming Frog SEO spider crawl of my site as a Googlebot user agent to see if there was some reason that… Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by bijutoha … Screaming Frog SEO spider crawl of my site as a Googlebot user agent to see if there was some reason that… What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by Dani In Google Search Console, I see a giant spike in "Other agent type" in crawl stats a few days ago. We don't use Google Ads (haven't for many years), and didn't add any large amount of content recently that I could imagine would constitute Google AdSense crawl. What could this be attributed to? Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by bijutoha If you’re not running **Google Ads** or producing a large volume of new content, chances are it’s not **AdsBot** or **something new being discovered**. Unless you're experiencing significant issues with your site, it’s usually nothing to stress over and could simply be background activity from Google. I honestly think sometimes Google's little … Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by Dani > Unless you're experiencing significant issues with your site Oh, but I **am** experiencing significant issues with my site. I also just noticed that within 72 hours of the spike, 80% of our URLs have been deindexed by Google. This is the first time this has ever happened in our 23 years. Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by bijutoha > > Unless you're experiencing significant issues with your site > > Oh, but I **am** experiencing significant issues with my site. I also just noticed that within 72 hours of the spike, 80% of our URLs have been deindexed by Google. This is the first time this has ever happened in our 23 years. Oh wow, Dani, that **80% deindexing… Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by Dani No manual actions or security issues. Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by bijutoha > No manual actions or security issues. Okay, Dani, that's genuinely good news about no manual actions or security issues, at least that rules out the scariest stuff! But it makes the 80% deindexing even more puzzling. Since it's not a penalty, my mind immediately jumps to some kind of technical configuration problem that might have … Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by Dani I posted a question over in [Google's webmaster forums](https://support.google.com/webmasters/thread/348804021/all-urls-being-deindexed?hl=en&sjid=2216033765847194048-NC) for some more details. As a status update, the mass deindexing seems to be happening on a large enough scale, across enough sites, that Barry Schwartz [wrote an article about … Re: What are "Other agent type" Googlebot types in GSC crawl stats? Digital Media Digital Marketing Search Engine Strategies by bijutoha > I posted a question over in [Google's webmaster forums](https://support.google.com/webmasters/thread/348804021/all-urls-being-deindexed?hl=en&sjid=2216033765847194048-NC) for some more details. As a status update, the mass deindexing seems to be happening on a large enough scale, across enough sites, that Barry Schwartz [wrote an article… Re: SEO vs AEO – Which is more important for the future of search? Digital Media Digital Marketing Search Engine Strategies by Dani … in Google's search results, we are required to allow Googlebot to scrape all of our content and use it for… Re: Lots of new members but no one posts Community Center Meta DaniWeb by Dani … go there. (In fact, we block known crawlers such as Googlebot with an HTTP 403 error, just in case any decide… Re: Google Search Console -> Page indexing Digital Media Digital Marketing Search Engine Strategies by ashleydent4u It sounds like you're on the right track already, and your robots.txt file actually looks okay at first glance. However, if 1.13K pages aren’t being indexed, it could be due to a combination of issues like low-quality content, duplicate pages, or crawl budget limits (especially with Blogger sites). In Search Console, head to the “Pages” section … Googlebot crawling AMP pages Digital Media Digital Marketing Search Engine Strategies by Dani Googlebot is crawling my AMP pages more than they are crawling … Googlebot?? Digital Media Digital Marketing Search Engine Strategies by iTaChi i checked my site's statistics..and i viewed the bots who visited my site.. then there was googlebot..40 hits..last may 9...but when i checked my index status in google..they said that my site is not yet indexed... the problem is..since my site was crawled by googlebot...why is it not yet indexed by google??? thank you... Googlebot: Valid meta robots content values Digital Media Digital Marketing Search Engine Strategies by Lampard33 Valid meta robots content values Googlebot interprets the following robots meta tag values: * …from being included in the index. * NOFOLLOW - prevents Googlebot from following any links on the page. (Note that… different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.) * NOARCHIVE - prevents … Googlebot gets to grip with Flash Digital Media Digital Marketing Search Engine Strategies by happygeek … Flash indexing is heading for Google. In fact, if the Googlebot engineers posting to that blog are to be believed, the…. There are some other limitations, according to the Googlebot engineers, namely: [LIST=1] [*]Googlebot does not execute some types of JavaScript. So… Googlebot ignores robots.txt Digital Media Digital Marketing Search Engine Strategies by Dani … not respecting my robots.txt. I'm seeing Googlebot's user agent crawling pages that have been in my … dates indicated as recent as yesterday. Additionally, I'm seeing Googlebot crawl my robots.txt file a few times a day… Re: Googlebot ignores robots.txt Digital Media Digital Marketing Search Engine Strategies by rproffitt I'm hearing more about fake googlebot requests. Maybe that? PS. Google those 3 words to find out more. Also: https://support.google.com/webmasters/answer/80553 Re: Googlebot ignores robots.txt Digital Media Digital Marketing Search Engine Strategies by Dani Per the OP post: > Additionally, I'm seeing Googlebot crawl my robots.txt file a few times a day, **and the URLs are definitely blocked per the Google robots.txt tester**. Re: Googlebot ignores robots.txt Digital Media Digital Marketing Search Engine Strategies by rproffitt Time for Google's own to give up ideas. Given the Google Seach Console (GSC) gives it a passing grade tells me that that likelihood of fake googlebot requests just went up even if the useragent is legitimate. Sorry for not defining GSC first. I'll work harder on that. Re: Googlebot ignores robots.txt Digital Media Digital Marketing Search Engine Strategies by Dani … rather large, and blocks a significant chunk of our pages, Googlebot may think that we made a mistake and didn't… Googlebot found an extremely high number of URLs on your site Digital Media Digital Marketing Search Engine Strategies by Dani … a message in my Google Webmaster Tools saying that "Googlebot found an extremely high number of URLs on your site… Re: Googlebot found an extremely high number of URLs on your site Digital Media Digital Marketing Search Engine Strategies by zabalex … for the pages or directories as per the requirement for googlebot to follow the instructions. regards zabalex Googlebot uses cookies Digital Media Digital Marketing Search Engine Strategies by Dani Google has always said that all of their bots do not use cookies. I caught a screenshot on Google PageSpeed Insights that displayed a modal that I only have implemented for users with a specific cookie already set. So there you have it, Googlebot uses cookies. Re: Googlebot uses cookies Digital Media Digital Marketing Search Engine Strategies by AussieWebmaster From 2014 https://www.tomanthony.co.uk/blog/googlebot-accepting-cookies/ Re: Googlebot help. Digital Media Digital Marketing Search Engine Strategies by Chuck's I have something new to me Googlebot-o, it has consumed a day yesterday and is still there today! Will it leave my stat counter soon? Thnak You:) Chuck Re: Can you tell me something about googlebot? Digital Media Digital Marketing Search Engine Strategies by Nicoli Googlebot is the name of google's search engine crawler/spider. It crawls to find new and updated webpage to index to Google. Re: Googlebot?? Digital Media Digital Marketing Search Engine Strategies by uggur Google has fresh bot and deep bot. Fresh bot coming to your site everyday. (Its change pagerank or other google criteria) Deep bot is coming to your site every month and its indexing your site to google database so you should wait a period. Re: Googlebot?? Digital Media Digital Marketing Search Engine Strategies by cooluks [QUOTE=uggur;365963]Google has fresh bot and deep bot. Fresh bot coming to your site everyday. (Its change pagerank or other google criteria) Deep bot is coming to your site every month and its indexing your site to google database so you should wait a period.[/QUOTE] Excellent explanation.. Thanks for this informative post...