Member Avatar for Dani

We had an issue with our load balancer yesterday which resulted in one of our six web servers sending 404 responses for all requests.

Obviously traffic was way down during the outage, but there was no significant recovery after the outage ended. I'm worried that googlebot came to crawl us and our servers essentially said, "Nope, sorry, those pages don't exist here," and googlebot, with no reason to think our servers were lying, deindexed us and/or stopped sending traffic to any page it got a 404 from.

What can I do to remedy the situation before it escalates further?

  • 3 Contributors
  • 3 Replies
  • 299 Views
  • 15 Hours Discussion Span
  • Latest Post Latest Post by bradly.spicer

Dani AI

Generated

Brief summary and immediate priorities: 's load‑balancer incident (one of six servers returning 404) creates exactly the "mixed signals" that can slow crawling, but short, isolated outages rarely cause permanent deindexing. The immediate objective is to (a) confirm the scope and timing of Googlebot hits during the incident, (b) get all servers reliably returning the correct status codes, and (c) prompt Google to recheck the highest‑value pages.

Quick triage (logs + Search Console):

  • Check access logs to find Googlebot requests that returned 404 and list affected URLs. Example commands (adapt to local log format):
zgrep -i 'googlebot' /var/log/nginx/access.log* | grep ' 404 '
awk '$9==404 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
  • Verify Googlebot authenticity with reverse/forward DNS (dig -x <ip> then dig +short <hostname>).
  • As suggested, use Search Console (Webmaster Tools) to inspect affected URLs, mark crawl errors fixed, and request reindexing for priority pages. 's crawl‑rate tweak can help in some accounts but is not a substitute for ensuring correct server responses.

Short recovery + prevention:

  • If a site or server is temporarily unavailable, serve 503 + Retry‑After instead of 404 so crawlers know it is transient:
HTTP/1.1 503 Service Unavailable
Retry-After: 3600
  • Fix load‑balancer health checks so they hit a stable health endpoint (eg. /healthz returning 200) rather than a content page that might return 404 during partial failures.
  • Resubmit sitemap, prioritize manual indexing requests for top pages, and monitor Search Console impressions and index coverage over the next few days. If index/ranking loss persists beyond a couple of weeks, deeper log analysis and configuration review is warranted.

Recommended Answers

All 3 Replies

Member Avatar for mmcdonald

Are you not able to get this sort of information from webmaster tools? perhaps request an increase to your crawl rate for a few days?

Member Avatar for Dani

GWT is always a few days behind, so it hasn't showed up there yet.

The problem with crawl rate is that, like it or not, Googlebot received mixed signals from the site, so now they are less likely to "trust" that site moving forward, me thinks.

Member Avatar for bradly.spicer

In GWT if you wait for the page errors to come up you can remove them all and Google will automatically re-crawl the site. This should negate any errors ^^

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.