Member Avatar for savedlema

Hifriends!
I've just started working on SEO for my sites, but I have problems in my first attempt.
I signed int to Google Webmaster tools...added a site and verified it successfully. My site even started appearing on Google Search and within a very short time, it was on the first page results!

But then it disappeared from Google Search.Even if I search for site:kilivacation.com its not showing up, Google says there is no such a site..if you are owner....

Going into Google Webmaster tools, if I go to Crawl -->robots.txt Tester, I get the error "You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file"

If I look at the sitemap (still in Google Webmaster) I now see an error "Network unreachable: DNS issue"

I also see Crawl Errors "Network unreachable: DNS issue"

Fetch as Google says "Status: Temporarily Unreachable"

My site is www.kilivacation.com.

Site shows up in the first page results of Bing with no problem.
How can I solve this Google Search problem?
Thanks for any inputs.

Edited by savedlema because: added the "fetch as Google" result
  • 6 Contributors
  • 9 Replies
  • 818 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by savedlema

Dani AI

Generated

Short, practical checklist and next steps for the robots.txt / DNS errors described by . The symptoms you reported are almost always a resolution or hosting-level reachability problem (intermittent authoritative DNS failures or a webserver that times out when Google requests /robots.txt). The replies from , and pointed in the right direction; below are concrete diagnostics and actions that were not covered in the thread.

Run these quick tests from your machine (replace the domain with your site) to see where the failure is occurring:

dig +trace kilivacation.com
dig @8.8.8.8 +short kilivacation.com A
curl -I --max-time 10 http://kilivacation.com/robots.txt
curl -I --max-time 10 http://www.kilivacation.com/robots.txt

What to look for and what to do next:

  • If dig returns inconsistent answers, SERVFAIL or no A record from some resolvers, you have DNS instability. Ask the host/registrar to check authoritative name server health and replication. Give them the exact Search Console timestamps so they can correlate logs.
  • If curl times out or returns 5xx for /robots.txt, check your webserver access/error logs for requests from Googlebot and for slow backend processes or PHP timeouts that might coincide with those requests.
  • Check firewalls, WAF rules, or security modules that could be rate-limiting or blocking Googlebot IPs; if using Cloudflare or similar, ensure "under attack" modes are not blocking crawlers.

Preventive and recovery steps:

  • Keep at least two reliable authoritative name servers on different networks. Lower TTL only while actively making DNS changes, then raise it again.
  • Implement a permanent (301) redirect to a single canonical hostname (as advised). If you only have cPanel access, use cPanel > Domains > Redirects to set a site-wide 301 from the non‑canonical to the canonical hostname.
  • Verify both host versions in Search Console (or the domain property if available), request an index for the homepage after fixes, and enable simple uptime/DNS monitoring so transient issues are caught quickly.

When contacting support, include the exact Search Console error messages and timestamps and the output from the dig and curl checks above.

Recommended Answers

All 9 Replies

Member Avatar for almostbob

it can be as simple as propogation time, DNS servers take time to react to the construction of new sites, or the changing of site structure.
very likely the links may simply not have reached google yet

sitemap submission to the webmaster tools page will increase depth of googlebot access

commented: Thank you very much. I'll see if more time solves DNS issue. +2
Member Avatar for Member #1111776

This has also happened to me when I had temporary DNS issues. Check your DNS status again in WT and look at the timeline. You can easily see how many errors and for how many days it lasted.

Your site now shows in Google.

BTW, I noticed your site displays with both the www and without. You should choose one and set a redirect either in your httpd.conf file (if you have access) or in your root .htaccess file.

commented: Thank you very much. Can you guide me on the redirect part? what is it and what is it good for?Thanks. +2
Member Avatar for Kelly Burby

Well, I wonder why you are unable to see your site over the google I have tried searching and it's showing your site over the search results.

commented: Thank you very much for offering to help. +2
Member Avatar for savedlema

Hi all!

Thank you all for your inputs. Yes, I'm surprised now the site shows up in Google first page results (key:. My last attempt was to remove in my Google Webmaster Tools dashboard and adding it up again. Glad now it shows up. But I have still been receiving daily emails from Google saying

" Googlebot can't access your site
Over the last 24 hours, Googlebot encountered 18 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 64.3%."

But at least now I have 64.3%, I used to have 100%! I hope it stays there and get better.

HappyCat, more help please with the last part of your comment, how do I do the redirect? What is it good for? (My only access is through cpanel so I think all I have is the .htaccess)

Thank you everyone.

Member Avatar for almostbob 
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.
RewriteRule ^(.*)$  [R=301,L]
  1. turns on rewrite
  2. if the link does not start with www
  3. replace link with www and 301 permanently moved

for SEbots it transfers most of the non-www link's mojo to the www version

Inform google directly in your webmaster tools panel

Edited by almostbob
commented: Thank you very much. I was able to read about this and implement. Now it works and I hope for the best. Thanks. +2
Member Avatar for PixelatedKarma

; go into your Google Webmasters Account and click on the left where it says "Crawl", then select the subcategory "Crawl Errors"; it should tell you exactly the pages that Google is having problems crawling.

As for the DNS problems, I can't add much more value to solving your problem then what others above have suggested. However I would suggest that you discuss the issues with your webhost. To me I would assume if Google is having problems with your DNS servers something is going on with your hosts domain name servers.

commented: Thank you very much Pk! +2
Member Avatar for Kelly Burby

Well, @friends this question has already been marked as solved so I don't think that there is any need for further comments.

Member Avatar for happygeek

It may have been marked solved, but in thir last post the OP stated "more help please" which suggests that there is, indeed, a need for further comments...

commented: Thank you mod. I decided to mark it unsolved for a while. +2
Member Avatar for savedlema

almostbob, Thank you very much. I was able to read about this and implement. Now it works and I hope for the best. Thanks.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.