304 Not Modified Crawl Requests

[Dani]

Starting September 1st, about 25% of crawl requests started returning 304 Not Modified instead of 200 OK. Prior to that, there were just a small handful of such requests per day.

I did not make any code changes that would introduce returning this header. Once I noticed, however, I tried visiting a handful of the pages as a logged-out user (where caching is enabled) and noticed that my Chrome web browser was also returning 304 headers.

We use Cloudflare. Did something change either with Googlebot (making the request) or with Cloudflare (responding to the request) that would cause this change of behavior? I found no articles suggesting that Googlebot just recently started using the if-modified-since header.

Edit: I'd like to add that I have Cloudflare configured to cache HTML pages for logged out users.

[AussieWebmaster]

you may want to try removing the caching code you added and should fix it - the instructions in 304s screw with browsers especially older ones

[Dani]

Hey Frank! :)

It's not caching code. I am simply returning a Cache-control HTTP header (specifying the page is cacheable by the web browser), same as I've done for the past 20 years.

[rproffitt]

I doubt that it's connected but I still see that bug with the spinning circle.

1. Go to https://www.daniweb.com/articles/latest/articles
2. Press the End keyboard button.

[Dani]

Yeah, that’s unrelated. Just a UI but I was planning on fixing snd then got caught up with other things and completely forgot about it. Can you please refresh my memory and link me to the thread where you wrote reproducible steps?

[rproffitt]

I don't think that's it either. But it's all of the 2 steps I wrote above. The old(?) thread is at https://www.daniweb.com/community-center/meta-daniweb/threads/535848/the-end-shows-when-reading-latest-daniweb-articles-on-chrome-and-opera

[Dani]

May I ask where you got to that URL from because there shouldn’t be a link anywhere on the site to that specific URL? If there is, it’s a bug.

[rproffitt]

Sorry if this belongs in that other thread:

1. I click on https://www.daniweb.com/articles/latest/recommended
2. The Oops screen comes up.
3. The Continue button link is https://www.daniweb.com/articles/latest/articles

Remember the spinning The End. will not occur in this sequence. Look up to see the sequence to get the spinning The End.

[Dani]

Gotcha!! Much thanks :)

[ET]

The HTTP 304 not modified status code indicates that the website you're trying to access hasn't been updated since you last visited it. In most cases, your browser will save (or cache) online pages so that it does not have to download the same information many times. This is an attempt to increase the speed with which pages are delivered.

[HelgaRaymond]

An HTTP 304 not modified status code means that the website you're requesting hasn't been updated since the last time you accessed it. Typically, your browser will save (or cache) web pages so it doesn't have to repeatedly download the same information.

[Dani]

Yes, of course I know what a 304 is. No need for everyone to regurgitate Wikipedia.

[Dani]

This is a very old Google article from 2008, but it gives a definitive answer that Google does indeed prefer those 304s, so I don’t have anything to worry about:

https://developers.google.com/search/blog/2008/11/date-with-googlebot-part-ii-http-status

Open to any more recent official stances on the matter. :)