Member Avatar for Dani

Starting September 1st, about 25% of crawl requests started returning 304 Not Modified instead of 200 OK. Prior to that, there were just a small handful of such requests per day.

I did not make any code changes that would introduce returning this header. Once I noticed, however, I tried visiting a handful of the pages as a logged-out user (where caching is enabled) and noticed that my Chrome web browser was also returning 304 headers.

We use Cloudflare. Did something change either with Googlebot (making the request) or with Cloudflare (responding to the request) that would cause this change of behavior? I found no articles suggesting that Googlebot just recently started using the if-modified-since header.

Edit: I'd like to add that I have Cloudflare configured to cache HTML pages for logged out users.

Edited by Dani
  • 5 Contributors
  • 12 Replies
  • 229 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

A likely explanation for the sudden jump in 304 responses is a change at the CDN/edge rather than a new bug in your origin code. Cloudflare announced and rolled out "Smart Edge Revalidation" in mid‑2021; it will synthesize revalidation headers when the origin doesn’t provide them and answer conditional requests at the edge, which increases the number of 304s returned to browsers and crawlers. (blog.cloudflare.com)

How that shows up in practice: when a client (browser or Googlebot) sends If-Modified-Since or If-None-Match, Cloudflare’s edge cache (and Workers cache API) can validate an existing edge copy and return 304 Not Modified without fetching a full body from origin. Cloudflare also exposes CF-Cache-Status values such as REVALIDATED so you can tell when a response was served after revalidation. That matches your observation that Chrome and Googlebot began getting more 304 responses. (developers.cloudflare.com)

Quick troubleshooting checklist (do these on a sample URL that used to return 200 and now shows 304):

  • Inspect response headers in DevTools or with curl and look for CF-Cache-Status, Age, ETag, Last-Modified, and Cache-Control.
  • Simulate a conditional request to provoke a 304 and see who answers it (edge vs origin): 
curl -I https://example.com/path
curl -I -H "If-Modified-Since: Tue, 01 Sep 2021 00:00:00 GMT" https://example.com/path
curl -I -A "Googlebot/2.1 (+http://www.google.com/bot.html)" -H "If-None-Match: \"<etag>\"" https://example.com/path

A CF-Cache-Status: REVALIDATED or a 304 with Cloudflare headers means the CDN did the validation. Use the Cloudflare cache response docs while you inspect. (developers.cloudflare.com)

What to do next: if 304 is acceptable, no code change is needed—304 is efficient and Google supports conditional caching. If you want fewer 304s, options include: increase Browser Cache TTL so clients revalidate less often; change Cache Rules / Edge TTL or disable HTML caching for those URLs; or make the origin explicitly emit ETag/Last-Modified and purge the Cloudflare cache on updates so revalidation behavior is clearer. Check Cloudflare’s Cache Rules and Edge/Browser TTL settings to implement the change that best fits your workflow. (developers.cloudflare.com)

Notes tied to thread posts: ’s suggestion to remove caching “code” isn’t necessary here if you only set Cache-Control headers — the CDN is the component now doing more revalidation. ’s Cloudflare HTML caching for logged‑out users is exactly the place to inspect CF-Cache-Status and TTL settings.

Recommended Answers

All 12 Replies

Member Avatar for AussieWebmaster

you may want to try removing the caching code you added and should fix it - the instructions in 304s screw with browsers especially older ones

Member Avatar for Dani

Hey Frank! :)

It's not caching code. I am simply returning a Cache-control HTTP header (specifying the page is cacheable by the web browser), same as I've done for the past 20 years.

Member Avatar for Dani

Yeah, that’s unrelated. Just a UI but I was planning on fixing snd then got caught up with other things and completely forgot about it. Can you please refresh my memory and link me to the thread where you wrote reproducible steps?

Member Avatar for Dani

May I ask where you got to that URL from because there shouldn’t be a link anywhere on the site to that specific URL? If there is, it’s a bug.

Member Avatar for Dani

Gotcha!! Much thanks :)

Member Avatar for ET

The HTTP 304 not modified status code indicates that the website you're trying to access hasn't been updated since you last visited it. In most cases, your browser will save (or cache) online pages so that it does not have to download the same information many times. This is an attempt to increase the speed with which pages are delivered.

Member Avatar for HelgaRaymond

An HTTP 304 not modified status code means that the website you're requesting hasn't been updated since the last time you accessed it. Typically, your browser will save (or cache) web pages so it doesn't have to repeatedly download the same information.

Member Avatar for Dani

Yes, of course I know what a 304 is. No need for everyone to regurgitate Wikipedia.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.