On Tuesday, November 15th, the average response time (ms) in the Crawl stats section of my Google Search Console skyrocketed. It's been slowly coming back down since, but I was wondering if something changed outside my control that may have caused this anomaly. I'm still working on getting to the bottom of it, and was just curious if anyone else has experienced the same thing around the same time?
Dani 4,329 The Queen of DaniWeb Administrator Featured Poster Premium Member
Alisha_8 34 Newbie Poster
It's possible that this anomaly could be due to a number of factors, including a change in the way websites are being crawled, an increase in the number of web pages being crawled, or the introduction of new algorithms by Google. It's also possible that this could be due to a technical issue on your end, such as a slow server or a bad connection. It's hard to say for sure without more information, but if you are still having trouble with the issue, it might be worth looking into what could be causing the issue and making any necessary changes.
Dani 4,329 The Queen of DaniWeb Administrator Featured Poster Premium Member
Unfortunately it still hasn't come back down entirely, and I don't have any ideas as to what could have caused it. There haven't been any big site restructuring or anything that would cause Google to suddenly start crawling my site completely differently. While temporary server downtime might lead to Google crawling a site a little less for awhile, the average response time should bounce back once the server is back online. (I haven't had any downtime though at all.) I also am pretty sure Google has acknowledged that algorithm updates are completely independent from crawl behavior. Either way, average response time isn't a crawl behavior, not directly anyways. Out of ideas for now. Has anyone else experienced anything like this around the same time I did?
Dani 4,329 The Queen of DaniWeb Administrator Featured Poster Premium Member
So I got to the bottom of it. I had accidentally adjusted the crawl rate in Google legacy site settings. Crawl rate is climbing back up now and average response time is making its way back down.
Dani 4,329 The Queen of DaniWeb Administrator Featured Poster Premium Member
I also want to point out that, with the huge decrease in crawl rate, the number of sitemap pages that were flagged as Discovered - currently not indexed was creeping up as well. It's finally working on decreasing again.
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.