What is the difference between indexed and crawling?

[Zinavo]

What is the difference between indexed and crawling?

[shivya bhargava]

Hello Zinavo,

Google crawl every page of your website but indexes only sitemap's pages.

You can restrict the google bots to crawl any page using robots.txt.

[Dani]

There actually were some more timely answers but they were all deleted for spamming. Better late than never.

However, since Shivya is not exactly correct, I'll elaborate.

Google crawls the web, following links, looking for pages with good content. As mentioned, you can restrict bots to not crawl specific pages based on your robots.txt file. You can recommend which pages of your site you care the most about in your sitemap file. Google crawls your site based on a crawl budget. Based on how much clout your site has in terms of incoming backlinks, your site's speed, server response time, and many other factors, Google determines a unique crawl budget for each site, that dictates how deep into your site they'll crawl. When they crawl, they read and understand all the content on each page.

Once they've crawled your site, only pages that make the cut make it into the Google index, where they become searchable. Duplicate content, low quality pages, thin pages, spammy pages, etc. all won't make the cut. If you have too many of those kind compared to all of the pages that Google wasted their time crawling, you'll be hit with a Panda penalty.