1.11M Members

StackOverflow stole all our search traffic!

 
5
 

As many of you know, DaniWeb was hit by a Google algorithm update back in November 2012 and we lost about 50% of our search traffic. In investigating the issue, I discovered that DaniWeb, in addition to most other programming forums out there, all lost their google traffic to StackOverflow.

For those who don't know, StackOverflow is a Q&A-based wiki-forum hybrid that was born a few years ago and subsequently received a lot of venture capital. Hands down, it deserves really high listings in Google results: it is fast-loading, lots of people link to it, it has almost exclusively high quality content, and their unique Q&A platform ensures that the best answers are always shown immediately below the question, optimized for those Google searches where you just need to find a quick answer without having to wade through discussions. All in all, no complaints. It deserves to rank high in many cases.

However, back on November 20th 2012, DaniWeb, and many other programming forums, all lost 50% or more of our traffic literally overnight. We were happily receiving about 300,000 pageviews per day for quite some time until we received 330,000 pageviews on Monday, November 19th. We then instantly took a hit down to 230,000 the next day. By Friday the 23rd, we were at 182,000 pageviews. It's been slowly going downhill ever since, and today we're down to averaging a mere 130,000 pageviews daily. I have heard very similar stories from many other programming forums. For example, the programming forum DreamInCode was hit at the exact same time as us by a similar percentage of traffic, as told by their Quantcast report.

Here's a graph of our organic traffic that shows the sudden decrease and subsequent decline:

2d117659370f038ade0072ad0e2a3296

What makes this story interesting is that at the exact same moment as every other programming community on the web lost their traffic, StackOverflow increased its own traffic proportionally. Within the month, at least according to Quantcast, they went from averaging 2 million uniques per day in October 2012 to 2.7 million in January 2013 and 3.2 million in March 2013.

It gets even more interesting, however, when you compare the traffic decrease that we received from Google organic SERPS to the traffic increase that we received from referrals from StackOverflow! The following graph shows that at the exact same time as traffic from Google to DaniWeb decreased, traffic from StackOverflow to DaniWeb increased by proportional percentages. In other words, we lost half of our traffic from Google, but we doubled the amount of traffic we were receiving from StackOverflow at the exact same time, practically down to the day.

11a818b5e1bb0475c42dbdc955f2e635

Upon close investigation, it seems that Google is replacing StackOverflow with DaniWeb for most of what we used to rank for. However, it's actually DaniWeb that has the better answers in certain circumstances, and so StackOverflow is actually linking to DaniWeb. What this means is that instead of just linking to the answer on DaniWeb, Google is now linking to StackOverflow who is linking to the answer on DaniWeb. How is this good for Google end-users exactly??

Here are some specific examples:

  • Example 1: A blatant link to DaniWeb that directly answers the specific question posed on StackOverflow is the most voted-up answer of all the answers provided. A Google search for the question has StackOverflow in three of the top results, and yet none of the three results even have a direct answer to the question. In fact, the second result was a closed and down-voted question with no answers at all. In Google's defense, DaniWeb did make it to the SERPS here as well, but it's clearly in StackOverflow's favor.
  • Example 2: Just like in the previous example, the most upvoted answer is a link to where the question was answered on DaniWeb. A Google search gives the StackOverflow page that points to DaniWeb for the answer #1 in the SERPS while that exact DaniWeb page containing the answer directly is #9 in the SERPS.
  • Example 3: This example couldn't be any more cut and dry. DaniWeb is linked to in the most upvoted answer provided, and a Google search gives StackOverflow the top three SERPS results (with the #1 result being a link to find the answer on DaniWeb!) with DaniWeb itself no where to be found in the SERPS.
  • Example 4 is just a blatant link to DaniWeb for the answer with DaniWeb no where to be found in StackOverflow-ruled Google SERPS
  • Example 5 is just a blatant link to DaniWeb for the answer with DaniWeb no where to be found in StackOverflow-ruled Google SERPS

And that's just the very tip of the iceberg.

C'mon Google, really? If I were into conspiracy theories, I might just think that because so many Google employees use StackOverflow themselves, they've tipped the algorithm to work in StackOverflow's favor in nearly all circumstances.

Want proof? DaniWeb used to host one of the most active C++ communities on the web, bar none. Today, most of our top members are seasoned C++ professionals and even published authors. If I do a simple google search just for the broad keyword "C++" and filter the results to only include Discussions, StackOverflow owns nine out of the ten results on the page! They own the top five results for the broad keyword php and the same for java.

Don't get me wrong, StackOverflow is a great site and deserves to rank well. But they don't deserve to be the only site that deserves to rank well. Not everyone in this world is a StackOverflow junkie, and a single domain serving nine out of ten of the SERPS results is more than a tad biased IMHO.

On a slightly unrelated note, it was brought to my attention at some point in history that people felt like StackOverflow has a higher caliber audience than DaniWeb does, which is the reason it ranks so much better. Judge for yourself: DaniWeb's demographics vs StackOverflow's demographics

Google has created a monopoly here, and on top of StackOverflow receiving millions in venture capital, every other programming community is being slowly squeezed out of existence. StackOverflow deserves to rank, but so do so many other programming communities on the web. There's solid proof here that Google is pushing quality forum pages out of the SERPs in favor of StackOverflow pages that do little more than link to the pages they've replaced.

Discuss ...

Attachments
Member Avatar
Dani Horowitz

I run DaniWeb. I coded (and currently maintain) the site from scratch using PHP/MySQL/jQuery and grew it into a thriving community of over 1 million IT professionals and 10 million visitors a month, making it not only one of the largest social media websites in its niche, but one of the largest publications on the web to date. I sell advertising. I host annual events throughout Manhattan during Internet Week, Advertising Week and Social Media Week which each see an average of over 1000 attendees. I host a bimonthly meetup group for techies on Long Island and eastern Queens. I speak on panels at search engine conferences. I run a brick-and-mortar coworking community for the tech industry in Queens, NY. I multitask. I consider myself a platform creator and a community builder. I'm a super-geeky programmer with a passion for Internet marketing. I am not for hire.

CriticalError
Deleted Member
 
0
 

The first example your websie shows in #2 while Stack shows in #1. If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

 
2
 

The first example your websie shows in #2 while Stack shows in #1.

While I've made sure to ensure that I had personal results turned off when writing this article, Google results do fluctuate and also differ based on geolocation and other such things.

If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

We use a lot of different means of driving traffic, not just SEO. As you can read in this article, a significant amount of our traffic comes from other sites (such as StackOverflow!) that link to us. However, as with all Q&A discussion communities on the web (including StackOverflow, of course), SEO is the #1 traffic generator hands down. A huge hit from Google is crippling no matter which way you slice it.

However, the point of this article is not to complain that we were hit. It was to point out that all programming communities were hit at the same time and had all of their SERPs traffic replaced by StackOverflow, to the point of where StackOverflow owns nine out of ten of the google results for the broad keyword C++. Additionally, not only have we lost our traffic, but we lost our traffic to StackOverflow pages that do nothing more than link to the pages on our sites that they replaced.

 
1
 

The first example your websie shows in #2 while Stack shows in #1. If you cannot keep up with the huge costs (I guess you can't) I think you might need to invest in a little advertising and not just relie on SEO.

I believe Daniweb alreaduy invests in advertising.

I have noticed this, when I first began there were tons of results linking to Daniweb (which were often extremely useful in the end being the reason I registered) but now all I see is StackOverflow and a ton of useless pages. StackOverflow is a very good site as is Daniweb, but sites like Daniweb also need the traffic. The main thing StackOverflow lacks in my opinion is the discussion aspect and intimacy and familiarity between users.

 
0
 

Google seems to not care about true value they want homogenized content - in this case they are taking the curated over the original - bad form G

CriticalError
Deleted Member
 
0
 

What did the Panda update actually do, did it remove un-quality/uique content? If that is the case how come websites like w3schools still appear in top results..? They do have old content and quite short.

 
0
 

This is rather odd indeed... Looks to me as StackOverflow spent a lot of money to get favorable placement. I may be mistaken because search engine placement is not really my strong suit, it's not one of my suits at all actually (lol)...

Keep up the good work though.

 
2
 

From a strictly SEO perspective, StackOverflow has overall higher quality content than the forums they have replaced in the SERPs because they have fewer mispellings, grammar mistakes, and leet speak, than most other forums.

However, this is a double edged sword. StackOverflow is able to achieve this because they have ridiculously stringent rules and moderators manually correct all spelling/grammar mistakes and rewrite question titles, etc. Questions that aren't perfectly written and concise are immediately closed or rewritten. Many feel as if StackOverflow is over-moderated and therefore completely lacks a friendly, community feel to it.

From an SEO perspective, sure, their content is written in perfect sentences with perfect spelling and grammar. Googlebot eats that right up!! But many humans get offended having a site over-moderate what they contribute, and Google needs to understand that when it comes to forums and community discussions (especially global communities where English isn't always necessarily the poster's native language), grammar shouldn't be the end-all be-all major factor.

 
0
 

Looks to me as StackOverflow spent a lot of money to get favorable placement.

That's just the thing. You can't buy Google placements. Or at least, it's never been heard of before.

 
1
 

I'm starting to hate the whole new "convergence" and "personalized" strategies for google searches and others.

When I search for a coding issue, it is usually some obscure and very specific issue. It is true that SO often offers one or two reasonably related threads that can be useful to me, but very often there are gems to be found in more obscure sources, like less-prevalent forums, old bulletin boards, comment sections, personal pages from enthusiasts, mailing-list archives, or scholarly articles. I have noticed recently (roughly in the time frame you are referring to), that at least the top 5 results on google are almost always SO threads, and typically (for the kind of specific issues I look up), only one SO result is remotely related to the issue I looked up and the rest are unrelated or don't have any useful discussion in it. This is really annoying. I know that SO ranks high (deservingly) in general, but it should not mean that it should rank high for every specific search.

My impression is that the way it used to be, roughly-speaking, is that the top results are those that contain the best match for the search terms, with some conservative filtering out (or pushing down) of less reputable sources. Now, it seems to be reversed quite significantly, i.e., pull up the most reputable sources (most popular sites) and consider the search terms matching almost as a second thought. It is true that the way it was before (and even worse back in the old days (late 90s early 2000s)) there would always be a lot of junk popping up, like link-vessels and other content-duplicating sites, but they were quick and easy to ignore. Now, it's much less junk, but also much less variety.

On a related note, I feel that all this business of personalized searches is also annoying in the same way: I'm not searching the web to find things I already know about. I'd wish there was a button to reverse it completely (as in, "give the results that are as far removed as possible from what I've previously visited").

 
1
 

Wow Dani, I am sorry to hear this. I have been a member here for a few days. I do not post a lot, but I do frequent the forums. Personally I enjoy this forum more than Stackoverflow.

Looking at your examples, I believe it is the way the answers are linking to Daniweb.

Example 1 : Link Text is "Here"
Example 2 : Doesn't add up at all to me. DaniWeb should clearly be on top in this case.
Example 3: Is the entire link instead of link text.
Example 4: Another entire link instead of link text.
Example 5 : This one doesn't add up either.

I would contact Google and find out why these results are being displayed like this. Maybe they can give you some insight. As a last resort you could always block any referer you wish.
Eventually the linking would stop. I think this could possibly hurt even more short term, but increase traffic like it was in the future.

 
1
 

I feel that all this business of personalized searches is also annoying in the same way: I'm not searching the web to find things I already know about. I'd wish there was a button to reverse it completely (as in, "give the results that are as far removed as possible from what I've previously visited").

I agree with that sentiment completely. I remember reading something a while back where a Google engineer was talking about the "personalised" search features and mentioned that there are over 50 different metrics used to track user activity for this purpose, including IP address, geolocation and other data that is somewhat more difficult to strip from your browser headers.

I almost always use incognito mode for searches in order to remove as much personal data and history from skewing the results as possible, but it is still ridiculously localised. IMHO the only time this is useful is in searching for a local business such as "pizza" and in most other cases amounts to a form of censorship.

Dani, I found this site years ago via a Google search, but honestly can't say the last time I have seen DW in a result set - I do notice it has become a rarety. In fact my search results are serviced almost exclusively by a small set of websites including the stackexchange family, wikipedia and youtube - anything beyond that I have to really dig around for.

It annoys me to no end that certain aggregator sites containing no original content and often chain-linking to each other (until finally bouncing you back to SO pages you already read earlier in the search) still manage to place highly in the result set. I feel like ever since Google won the search engine war there has been a significant lack of competition (and no I don't think Bing really cuts it, sorry).

I don't really see the point in all their quality algorithms TBH. Most crap sites get weeded out in the long run by the internet community anyway, and short of actually hiring their own editors to curate the entire net they really can't guarantee accuracy or correctness of data at all. Wikipedia is case in point for this, yet somehow they are still viewed as some omniscient authority.

Anyway, sorry if this became a bit of a rant. But it does kind of make one long for the good ole days of simple keyword based search results.

 
1
 

My biggest gripe is that while they definitely deserve to rank well, they don't deserve to be the ONLY site that ranks well. Owning nine out of ten results for all discussion-based results with the broad keyword "c++" is a bit too extreme IMHO.

They seem to be ranking well NO MATTER WHAT, even in cases where they are simply doing little more than linking out to other sites that have the complete answer. The fact that Google is sending traffic to StackOverflow even in cases where the StackOverflow landing page doesn't contain the answer directly is proof that the algorithm is too biased towards the stackoverflow.com domain as a whole.

Google should be spreading the wealth around a bit more. Not everyone appreciates the Stack Overflow format, and Google is creating a monopoly here. IMHO no single broad keyword should ever be nearly completely owned by a single domain unless the keyword is a brand.

 
4
 

I dunno, in my mind, the concept of the search engine is that, at its core, it's supposed to be designed to give the end-user options of where to visit, given the user's specific criteria. Imagine looking up attorneys in the yellow pages and the first complete page of listings are just all different offices for the same law practice. Sure, they might win 90% of their cases, but that doesn't mean that I still don't want to be presented with options. Not everything is suited well for everyone.

CriticalError
Deleted Member
 
1
 

Well I agree with Dani, the results should be spread out, but since StackOverFlow is huge there are same / similar questions and I think this is the reason they get ranked more. If you type "c# xml writer" the #2 and #3 are StackOverFlow. When you type C++ stackoverflow does not show. Just remember one thing Dani, the results will be different depending on which country you're from. If you type something simple like HTML, or HTML tables then W3schools is likley to get top results. So I don't think Google has pointed everything there..If they did I'd expect w3schools to disappear as well..

 
2
 

Google is killing millions of small websites and favoring a few big websites. This is unfair. This will create monopoly in every field.

CriticalError
Deleted Member
 
-1
 

Google is killing millions of small websites and favoring a few big websites. This is unfair. This will create monopoly in every field.

Is there any solid proof?

 
2
 

Yes, there has been proof lately that Google's algorithm tends to significantly favor big, established brands. However, as with DaniWeb, it is possible for a small one-or-two person company to grow into a huge online brand without millions of dollars. Remember, having a huge online brand identity and huge online presence doesn't necessarily have to match up with what you would consider a big brand in the real world.

 
-6
 

The word "Stole" is pretty serious accusation. It implies that they have deliberately did something unethical.

I think the explanation is rather simple - Daniweb insults their users with popups in their face and flashing banners while SO just gives people what they want without all the annoying ads.

I don't beleive that this has anything to do with favoring big brand, I think it's that Google has a way to find what people preffer. When Google noticed that most people preffer answers from SO over almost all other programming forums then it started placing the SO threads even higher in search results.

Simple as that. If you want to compete with SO then stop whining and throwing the words "Stole" around and start concentrating on providing the best answers with the least amount of annoyances.

PS. I am going to inform SO about you accusing them of theft and hoping that they will remove all the links to Daniweb from their answers. I think it will be very fair answer to your unfounded and groundless accusations.

 
-2
 

Dani, you asked me a question in the commend, but I cannot reply to a comment. So I have to reply here:
You asked me "if you don't like our flashing banners..."
My questions is - how many people do you know that do like flashing banners and popups?
As for disabling banners in profile, I don't see the reason for that at all. if you have an option to disable banners in profile and you know that most users don't like banners, then why not just disable banners for every registered member and then have an option to enable banners - so those who love banners can enjoy them.

Isn't it about time forums rewarded their contributors?

Earn rewards points for helping others. Gain kudos. Cash out. Get better answers yourself.

It's as simple as contributing editorial or replying to discussions labeled or OP Kudos

You
This is an discussion and contributors may be rewarded
Post:
Start New Discussion
View similar articles that have also been tagged: