Member Avatar for ketanpatel643

We have a three year old website with a home page rank of 4. The site has over 4,000 pages, out of which only 1800 are now shown as indexed by Google.We tried to find out the reasons and after a lot of brainstorming we came up with some possibilities related with our own site as well as certain factors not in our direct control.

  • 14 Contributors
  • 14 Replies
  • 261 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by rowiki

Dani AI

Generated

A few practical steps to stop guessing and find the real cause (builds on points from , , and ): start by diagnosing with Google’s tools, then fix the highest-impact technical blockers first.

  1. Diagnose in Google Search Console — Coverage and the URL Inspection tool tell you the exact reason a URL isn’t indexed (examples: “Crawled — currently not indexed”, “Blocked by robots.txt”, chosen canonical, soft‑404s). Use those reports to prioritize fixes rather than submitting every page blindly. ()

  2. Quick checklist (do these in order):

  • Confirm the URL returns a proper HTTP status and isn’t accidentally noindexed or blocked by robots.txt. If you need to check headers quickly, run:
    curl -I https://example.com/path/to/page
  • Make sure sitemaps only list canonical, indexable URLs and split files correctly (50k/50MB limits). (sitemaps.org)
  • Verify rel="canonical" is set consistently (redirects + canonical = strongest signal). (developers.google.com)
  • Look for soft 404s and thin/duplicate content; these waste crawl budget and often cause “crawled but not indexed.” (searchenginejournal.com)
  1. Monitor crawl behaviour and server health: use the Crawl Stats / host status reports and your server logs to confirm Googlebot can fetch pages reliably (5xx spikes, long response times, or frequent soft 404s will reduce crawls). For larger sites, follow Google’s crawl‑budget guidance and remove low‑value URLs or noindex them so bots focus on important pages. (developers.google.com)

Practical workflow: pick 20 representative unindexed URLs → inspect them in GSC → fix robots/noindex/canonical/response-code issues → improve a few internal links or merge thin pages → request indexing for priority pages via URL Inspection (don’t mass spam requests) → recheck Coverage and Crawl Stats. Repeat until the excluded counts fall. ()

Note: sitemaps + backlinks help (as several posters said), but they’re secondary to fixing crawl/index blockers and content quality; diagnose first, then apply the targeted fixes above.

Recommended Answers

All 14 Replies

Member Avatar for karol33

is everything ok with the sitemap?

Member Avatar for digital29

build a sitemap, add it to Google Webmasters tools, build some backlinks for some inner pages and wait

Member Avatar for joelchrist

You need to work on all your web page links, like do social book marking, inner page directory submission, blogs and so on. It will helps google to index all you pages as soon as possible.

Member Avatar for AirForceOne

Sitemap submitting would not guarantee a thorough page indexing for your sites while deep backlinks to your inner pages would be really helpful for this purpose.

Have a nice day,

Member Avatar for InsightsDigital

Another reason can be if your site uses any Black Hat SEO. You have to make sure your site does not use any which affects why Google is not indexing your pages.

Member Avatar for jay 11

There's a lot of buzz about Google undergoing some major ranking and indexing changes - or a major "recrawl" of the web. Some webmasters have dubbed a recent change as the "Mayday" update.

I have personally seen a large dip in the number of pages Google has indexed for my site (which has millions of pages). Google maintains that the recent updates were purely algorithmic. However, I am seeing some low numbers of pages indexed from some pretty big sites. Either way, there definitely appears to be some shifting going on.

Member Avatar for eimroda

You can try any or all of the following:

1. Submit/create a sitemap to Google webmaster tools.
2. Create back links for your inner pages
3. Improve the link structure of your website- this will also improve the crawlability of your site.

Member Avatar for stellarios

try bookmarking those pages which are not indexed..

Member Avatar for mackone

Work on your content and getting links, especially deep links.

Member Avatar for jay 11

Work on your content and getting links, especially deep links.

I agree - improving overall content and maybe updating the internal link structure of a site, could force the spiders to re-visit and index.

Member Avatar for searchengineace

Sometime the poor internal linking also results in non indexing of pages on a site. Though all pages might have been indexed at some point of time, but after some time most inner pages might be left out. Besides having properly updated xml sitemap, one should work on ways to improve the internal linking of the pages. maybe an HTML sitemap linked to index page or group of pages references from various website pages, which are in search engine index. It should get you back on track

Member Avatar for jackalex05

I also puzzle about it.I do some submission on my site,but it useless.

Member Avatar for hardtie

Get deeplinks for the pages that are not indexed particularly.

Member Avatar for rowiki

Google index your page in time, not all at once, wait and you will see.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.