I am new to all this, as well as my site, and I would like to know if anyone can help me find out if my new website has been crawled.

I have not paid for any advertisement but submitted it to the free Google, Yahoo, and Open Diretory tools I found on the internet. Any help or advice would be welcomed

KM1

Dani AI

Generated

Quick primer and practical checklist. Crawling (a bot fetching a URL) is not the same as indexing (the page appearing in search results). As and touched on, a cached snapshot or a few indexed inner pages does not prove the homepage is indexable. Common causes for a homepage to be crawled but not indexed include an accidental noindex (meta or header), a rel=canonical pointing elsewhere, a redirect or www/https mismatch, robots.txt rules, or the page being treated as a soft 404 or duplicate content.

Basic diagnostic steps (run these from a terminal or ask the host to run them):

Check HTTP headers and follow redirects

curl -I -L https://example.com/

Look for a final 200 OK (or an unexpected 301/302/404/503) and any X-Robots-Tag headers.

Check for meta robots or canonical tags in the HTML

curl -s https://example.com/ | grep -i '<meta[^>]*name=["'']robots'
curl -s https://example.com/ | grep -i 'rel=["'']canonical'

A noindex meta or a canonical pointing to another URL will prevent indexing.

Verify robots.txt and sitemap

curl -s https://example.com/robots.txt
# look for "Disallow: /" or other blocking rules

Confirm the sitemap actually lists the homepage using the exact canonical form (same scheme and host: https vs http, www vs non‑www).

Other checks: ensure the homepage has substantive unique content (avoid thin pages that trigger soft‑404 behavior), confirm internal navigation links point to the canonical homepage URL, and review server logs for genuine Googlebot fetches (user‑agent + reverse DNS if needed).

Next steps: if headers/html look correct and the sitemap lists the homepage, request an index check with the site’s Search Console / Webmaster Tools URL inspection tool and monitor server logs for re-crawl. Tiebacks: as asked about meta tags and noted sitemaps, those are the first places to verify when inner pages appear but the homepage does not.

Recommended Answers

All 19 Replies

Type site:www.domain.com into Google and if any pages come up, then you're site is indexed.

I have tried this and nothing comes up in Google, or Yahoo for that matter. I only sent the URL in on the free submission forms from both Google and Yahoo, as well as the Open Directory Project. That is why I was asking if anyone could tell if my site had been crawled or could tell me a more specific way to find this out.

Also, sorry for posting my URL here. My ingorance for not reading through the FAQ's.

KM1

If nothing comes up after doing a site: query then you most likely are not in Google's index. To find out the details, including WHY you might not be included, sign up with Google webmaster tools at www.google.com/webmasters/.

I am new to all this, as well as my site, and I would like to know if anyone can help me find out if my new website has been crawled.

I have not paid for any advertisement but submitted it to the free Google, Yahoo, and Open Diretory tools I found on the internet. Any help or advice would be welcomed

KM1

Use this command cache:

If it displays that the site or page doesn't exists it means it is not yet index, so use site:mydomain.com..
Hope this helps....

No, that doesn't tell you if your site has been crawled. All those tell you if your site has been indexed.

Here's a tool that will tell you if your site has been crawled:

If it works, you'll see something like this:
"Googlebot last successfully accessed your home page on Nov 20, 2007"

That means your site has been crawled.

As far as being indexed, it takes time, but Google knows about your site if you've been crawled, and you just need to get out of the "sandbox" which can take several months. Try to get inbound links to boost your site's popularity in the meantime.

Or check the statistics of your website, and see whether you got some visitor from google.

Why don't you use your website as your signature here... you could be easily indexed by Search engines...

Or

my suggestion - place your website link to a web page which is already indexed by Search engines and in regular indexing.

I thought I could not use my website on this board? They snipped it out in my first message.

What do you mean to use my website as my signature?

KM1

OK just got crawled by googlebot late last week. Two of my pages have been indexed. In fact, when I type in my site in
google just like a normal search, becoming a true champion, it comes
up on the first page about #5. However, it is not my main page, which
is what I originally submitted 1 week ago. Why is my main page not
coming up? Why would google index 2 of my pages and not all 5,
including the default main page? Few of the key words that are on the
mainpage are listed, especially the main ones that are in the title of
the main page and listed as key words. Why is this?

Currently I am trying to get inbound and outbound links to my site.
This is a task in itself. Anyone know of another way other then
getting in contact with the site owner?


I am going to try and put my links on the main page (default page),
will this help on getting that page indexed and coming up in google
first before the other pages?


KM1

Have you filled in the META properley?

That I am not sure about. My webadministrator set up the whole site once I turned over the design. Is there a way you can view the source and let me know? I would have no idea. becomingatruechampion.com

In addition, I have yet to see anything on Yahoo. Submitted to them, through there free submission page, about a week ago also.

KM1

Your Site is new. Have you created a site map for your web if you not just create a xml site map and submit your site map using your Google Webmaster Tools account or you can submit them directly using addurl form of Google, for yahoo just w8 for some time yahoo is take some time to index the website but if you need some fast results than build some good quality links for your website and put your site in forums with in your signature.
Agrawat

Yes it is new. I have an XML site map already uploaded and accepted as viewed by google webmaster tools. Thanks though.

Could you explain what it means to put your site in your signature on forums. Many forums do not want you to promote sites on their boards. How do you put your site in your signature without getting it snipped out???

Yes, Yahoo has nothing on it yet. Google, as I said befor, has only 2 pages indexed and not the main page. Page 2 A Champion and 3 The Author. That is it.

KM1

No, that doesn't tell you if your site has been crawled. All those tell you if your site has been indexed.

How come you conclude to that?

Here's what we see when we the cache command:

This is G o o g l e's cache of http://www.yahoo.com/ as retrieved on 26 Nov 2007 03:02:04 GMT.
G o o g l e's cache is the snapshot that we took of the page as we crawled the web.
The page may have changed since that time. Click here for the current page without highlighting.
This cached page may reference images which are no longer available. Click here for the cached text only.
To link to or bookmark this page, use the following url: http://www.google.com/search?hl=en&q=cache%3Ahttp%3A%2F%2Fwww.yahoo.com&btnG=Google+Search

Try it and tell me that doesn't tell if site has been crawled.

So instead of logging in in to your Google account you can use the cache command...
When we use the site: command you can see how many are index.. I'm I right?

When you use the site: command you can see a selection of pages that Google has in their index. It isn't all of them. For that, you need Google Webmaster Tools.

This has been an extremely helpful post. Thanks to everyone who replied.

Sincerely,

Kevin

You can go to Marketleap.com. They have some good free tools so does Seomoz.org

Sign up for a Google account, and click under Webmaster Tools. Verify your site, and check the Overview section. Google will notify you if your site is indexed. For the other search engines, using the site: command is fine. Same for Google as well.

i have a similar problem. it has been over a month and google has not indexed my site

<snip url>

pls. help

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.