Member Avatar for devesh9392

Hi

On my website i have almost around 25000 webpages(static).
when i go to google and enter

inurl:mysite.com

it shows only 1000 pages. these are the pages crawled by google.
Here i notice that some of my webpages have never been crawled or never been indexed.
Also some of my webpages haven't been crawled since july 10, while some other pages have been crawled very recently(9 sept).

BTW it is PR 4 website.

can i do anything so that google crawls some more pages.

  • 3 Contributors
  • 7 Replies
  • 291 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by stymiee

Dani AI

Generated

A short, practical add-on to the existing replies from , and — focused on what to do next (not just why).

Don’t rely on inurl:/search-result counts to measure indexing. Google’s on-page result counts are estimates (and Google has recently made them harder to find), so they’re not a reliable way to count indexed pages. Use Google Search Console (GSC) instead — it’s the authoritative place to see which URLs are indexed and why some are excluded. (seroundtable.com)

Concrete steps to get more pages discovered and crawled:

  • Verify your site in Google Search Console and use the Page Indexing / Coverage reports to find which URLs are “Excluded” or “Crawled – currently not indexed.”
  • Build and submit XML sitemaps (use sitemap index files for very large sites and include only canonical, useful URLs). Sitemaps tell Google about your URL inventory; they don’t guarantee indexing but they help discovery. (developers.google.com)

Fix crawl‑efficiency and crawl‑budget issues:

  • Reduce low‑value/duplicate URLs (use rel=canonical or noindex where appropriate), block truly unneeded URL patterns with robots.txt, and speed up server responses so Googlebot can fetch more pages.
  • Monitor GSC’s Crawl Stats and your server logs to see real Googlebot behavior; popularity and perceived page value also increase crawl frequency, so prioritize pages you want indexed. (developers.google.com)

Meta tags and linking — what really matters:

  • Meta-keywords are ignored by Google; duplicate meta descriptions won’t stop crawling though they aren’t helpful. The meta robots noindex does prevent indexing — but Google must be able to crawl the page to see that tag (so don’t block it in robots.txt if you want noindex to work). Boost discovery by adding real internal deep links and getting a few natural external links to important inner pages. (searchenginejournal.com)

Quick checklist to apply now:

  • Verify GSC, submit sitemap(s), inspect a few representative URLs with the URL Inspection tool and request indexing for important pages.
  • Fix any server/HTTP errors, consolidate duplicates, improve internal linking to orphaned pages, and watch Crawl Stats/Indexing over the next few weeks. (developers.google.com)

This fills out ’s “deep links” idea with the practical GSC/sitemap/crawl‑budget steps, and clarifies ’s meta‑tag point with the noindex caveat.

Recommended Answers

All 7 Replies

Member Avatar for stymiee

The largest factor in determining how many are indexed are the PR of the individual pages. Not the PR of the homepage. You'll need to make sure you get deep links to your site as well as spreading the PR from your homepage around as well.

Also make sure all of your pages can be reached by the search engines. Just because you can get to them in your browser doesn't mean their crawlers can get to them.

Member Avatar for devesh9392

You'll need to make sure you get deep links to your site as well as spreading the PR from your homepage around as well.

Can you please tell me what do you mean by "deep links" ?
and how can i sprear pr from my home page ? (by linking all important pages to home page ? )

thnx.

Member Avatar for jbennet

Is the correct META tags present?

Member Avatar for stymiee

Can you please tell me what do you mean by "deep links" ?

A deep link is a link to an inner page in your site (e.g. not your home page).

and how can i sprear pr from my home page ? (by linking all important pages to home page ? )

You can spread PR by linking to inner pages. Cross linking your pages is an important part of on site SEO.

Is the correct META tags present?

Meta tags won't affect crawl rate. The most you can do is stop a page from being crawled and/or indexed.

Member Avatar for devesh9392

Meta tags won't affect crawl rate. The most you can do is stop a page from being crawled and/or indexed.

do you mean if i have same meta tag keywords and description for all my pages,
it would be considered as spamming and google won't crawl any further ?

Thanks...

Member Avatar for devesh9392

You can spread PR by linking to inner pages. Cross linking your pages is an important part of on site SEO.

does it mean i should have more than one link to reach any page in my website ?

Member Avatar for stymiee

do you mean if i have same meta tag keywords and description for all my pages,
it would be considered as spamming and google won't crawl any further ?Thanks...

No, it means that the meta robots tag should only be used to block content. It doesn't help it be crawled or indexed.

does it mean i should have more than one link to reach any page in my website ?

As long as it makes sense to have those links, then yes.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.