Member Avatar for crescendo

Which are the important aspects to keep in mind while creating a website that can help making a crawl able website? My previous websites are not crawled by Google. I have created those websites by using Wordpress.

  • 8 Contributors
  • 7 Replies
  • 195 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by Kelly Burby

Dani AI

Generated

Good summary from , and others on the initial steps. Once verification, a sitemap and the basic WordPress settings are handled, the remaining causes are usually at the server, header, or rendering level. The checklist below focuses on practical diagnostics and fixes that go beyond the usual “submit sitemap / verify” advice.

Quick diagnostics (replace example.com):

curl -s https://example.com/robots.txt

Look for Disallow: / or unexpected rules and confirm the sitemap location.

curl -I -L https://example.com/some-page/

Confirm final HTTP status (200 vs 3xx/4xx/5xx). Repeated 3xx loops, 403 or 503 will prevent indexing.

curl -I https://example.com/some-page/ | grep -i X-Robots-Tag

Some servers or CDNs send an X-Robots-Tag: noindex header that overrides page HTML.

Also compare raw HTML (curl https://example.com/some-page/) with the browser-rendered page to spot content that only appears after JavaScript runs; client-side–only content often needs server-side rendering or prerendering for reliable indexing.

WordPress-specific checks that aren’t always obvious: SEO plugins or maintenance plugins can set global noindex or return 503s; .htaccess or firewall rules can block crawlers; rel="canonical" tags may point to a different hostname or protocol; paginated or thin pages may be de-prioritized. Orphan pages (no internal links) won’t be discovered readily — server logs or a crawl tool will reveal them.

Where things still fail, server logs are decisive: search for genuine Googlebot requests (user-agent + access patterns) to confirm crawling attempts, then match those to the HTTP headers above. Use the URL-inspection / coverage reports in Search Console to see specific reasons Google reports for non-indexing. This complements the basic advice already raised by and and targets the less-obvious blockers that commonly stop WordPress sites from being crawled.

Recommended Answers

All 7 Replies

Member Avatar for mindstreak

Make sure you have submitted your website in google webmaster.
Integrate the sitemap and verify it in webmaster tools
Submit your website to various search engines.
Use the robots syntax in the header tag of the website.

Member Avatar for sumansingh

Atfirst submit your site on googlr webmaster and veryfy them. submit your site all search engine site.

Member Avatar for almostbob

As the two prior posters answered
Google will evetuaqlly find your site once an indexed site creates a link to it
which takes time
Get a
submit your xml sitemap there are wordpress plugins to create sitemaps directly from content.
then follow the directions given by google to solve problems in your site

Wordpress sites are usually easily indexed by google, but google has to know it exists

Edited by almostbob
Member Avatar for Ketul_1

First of all, verify your site on google webmaster. Write descriptive and unique content about product or services that page displays. Submit URL on different search rngine for crawl and index.

Member Avatar for rahulhog

If you are using wordpress make sure that you unchecked the option "Discourage search engines" on reading settings. Also make sure that robots.txt isn't block your important webpages. You can submit xml sitemap to google for better crawling. "Fetch as Google" option is available on google webmaster tools to crawl pages manually.

Member Avatar for Shabu Anower

While development keep in mind following things:
Search engine friendly URL's
.htaccess is not blocking search engine bot. Sometimes, wrong code may block your site from being indexed.

After development, do at least following thing:
Share your website in social network
Post a worthy comment in busy blog

If you follow them, you don't have to worry for indexing.

Edited by Shabu Anower
Member Avatar for Kelly Burby

Well, inorder to make your site more visible to the search engine you need to make sure that you have submitted sitemap in your Google webmaster tool, this will help the Google spiders in knowing where the things are placed. Once you submited the sitemap in the Google webmaster tool, use you fetch as Google option and submit your site for indexing.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.