Member Avatar for WoodiE55

I have some concerns I’m hoping someone can help me with. This time last year according to my servers web stats Google used about 5.86GB of bandwidth spidering my site, however now Google has only used about 8.8MB spidering my site. Just a few months ago - going to Google and typing in site:rcnitrotalk.com would return over 30,000 pages, however this month it returns 782 pages at the moment - a pretty big difference!

The only thing I can think of that's any different is I've moved the Google ads banners around. If google ad banners are the cause of my site not being displayed on Google anymore then I'd much rather remove the ads then not have any visitors to the site.

At any rate I'm really someone could give me some insight as to why my site was doing so well as being indexed to now hardly even seeing google on the site at all. The low search results has been going on now for about a month or two and seems to only be getting worse.


-WoodiE
www.rcnitrotalk.com

  • 2 Contributors
  • 3 Replies
  • 225 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by Dani

Dani AI

Generated

Quick diagnosis and practical next steps for the sudden drop from ~30,000 to ~782 indexed pages (and the big fall in Googlebot bandwidth). As noted, changes in how Google crawls a site can follow changes in external signals — but several technical problems produce the same symptoms. The reported “site:” count is only an estimate and can swing; the important thing is to confirm whether Googlebot is being blocked or the site is returning errors.

Helpful checks (run these and record results):

  • Verify robots.txt and HTTP headers for accidental blocks or an X-Robots-Tag: curl -I https://yourdomain.com/robots.txt and inspect for Disallow: lines.
  • Look for site-wide meta robots noindex or incorrect rel=canonical tags in templates. View page source or fetch headers with curl -L -s -D - https://yourdomain.com/page | sed -n '1,80p'.
  • Inspect server logs for Googlebot activity and status codes (large numbers of 4xx/5xx will reduce crawl). Example commands: 
    grep -i googlebot /var/log/apache2/access.log | tail -n 50
awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head
  • Check for duplicate/parameter URLs or session IDs that cause Google to collapse many variations into fewer indexed entries. Confirm sitemap presence and that submitted sitemap matches canonical URLs.
  • Scan for hacked pages or content-injection; compromised pages can trigger de-indexing or penalties.

Remediation priorities: remove any accidental Disallow or noindex, fix server errors and redirects (use 301 for moved pages), submit an up-to-date XML sitemap and use Google Search Console’s Index Coverage / URL Inspection to identify and request reindexing. Improve internal linking and restore/replace any high-value external links if they were lost.

A note on AdSense: simply moving ad blocks will not by itself remove pages from the index unless the change introduced cloaking, heavy client-side-only rendering, or accidentally altered templates to include noindex/blocking headers. Given the large bandwidth change reported by , log analysis and header checks are the highest-value first steps.

Recommended Answers

All 3 Replies

Member Avatar for Dani

It looks like your site is pretty dynamic, so that doesn't appear to be the problem. Could you have been banned for any reason? Are you using any black hat SEO techniques? (i.e. cloaking, etc?) How many other sites are linking to you? When I do a link: on Google, it doesn't look like very many at all. That is most likely your problem. A huge factor to how in depth google spiders your site and how frequently they return to your site, is how many other sites link to you.

Member Avatar for WoodiE55

csgal,

No I've never tried any black hat SEO techniques, the only thing I've really done has moved around the Adsense boxes and I've also redone a few things such as as header tags and such. Just making it better xhtml code, short of that nothing else.

As for other sites linking back to RCNT, what I don't understand is that I've never done any kind of advertising nor have I ever cross linked with other sites in the past yet I was still able to get 30,000+ pages indexed on google then all of a sudden I'm down to 700 pages. I also thought links to your page mainly effected your page rank not if your page gets indexed or not. Speaking of PR my front page is a 4 and my forum is a 3 - while not the best by no means I'd still say that says something doesn't it?

I dunno, I just don't understand why I was able to get over 30,000 pages on google with no hacks, links or anything else yet now I'm down to 700. :o

-WoodiE

Member Avatar for Dani

Getting incoming links is critical to good rankings. The more backlinks you have, the higher the pagerank - which means the more frequently google will respider your site, and the deeper they will crawl on each visit. (It doesn't influence your rankings, it just influences how many of your pages are spidered, which seems to be the problem.)

What could be possible is a particular reallllly high ranking page linked to you for a short period of time, convincing google that your site was worthy of frequent and deep crawling. But then that site removed you for some reason or another.

Either way, good luck! And get that backlink campaign started! ;)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.