Member Avatar for vivosmith

My URL is:
and for robots:

Hello,

When I started, I only submitted one site to Google (which went fine). Then I tried adding more pages, and it would not go through. I have added a phpbb script, tried editing it, to no avail. I tired messing around with eiliminating or reuploading my robots.txt file, no go. So what is the deal? I can reach it with my browser, but not with fetch or sitemap tool. I have 644 permissions on both, so that is not a problem. I would ask my host, but the only problem is that they are snail mail, and I need answers. So what's up? I checked everything, and it still won't work. Thanks

Vivo Admin,

  • 3 Contributors
  • 5 Replies
  • 252 Views
  • 4 Hours Discussion Span
  • Latest Post Latest Post by Member #949455

Dani AI

Generated

Short, focused diagnostic that builds on and : the browser can show a file while Google’s crawler can’t fetch it because of how the server responds or where the files live, not usually because of PHPBB or simple file permissions alone. The usual culprits (in descending frequency) are: robots.txt not at the domain root / wrong host or protocol, redirects or soft‑error pages, server-side blocking/firewall or mod_security rules, incorrect content‑type or encoding (including a UTF‑8 BOM), or an indexing header that tells bots not to index. RFC and search-engine guidance require robots.txt at the site root and plain text; sitemaps/text-sitemaps have simple UTF‑8 rules. (datatracker.ietf.org)

Quick, repeatable checks (run these from a shell on any machine that’s not your browser):

# fetch headers as Googlebot
curl -I -A "Googlebot/2.1 (+http://www.google.com/bot.html)" -L https://example.com/robots.txt
curl -I -A "Googlebot/2.1 (+http://www.google.com/bot.html)" -L https://example.com/sitemap.txt

Expect a direct 200 (no HTML error page), Content-Type of text/plain or application/xml (for XML sitemaps), and no redirects from www → non‑www or http → https that Google isn’t following. If these return 403/401/5xx or an HTML page, that explains the fetch failure; server errors and soft 404s prevent proper crawling. (developers.google.com)

Deeper checks and fixes: inspect webserver access logs for requests from Googlebot and the HTTP status shown; verify any firewall, fail2ban, or hosting security rules aren’t blocking Google’s IPs; confirm there’s no X-Robots-Tag: noindex header on responses (headers can block indexing even when a browser sees the file). Also save sitemap/robots as plain UTF‑8 without a BOM, and put any Sitemap: https://… lines in robots.txt if helpful. For X‑Robots‑Tag and meta rules, follow search‑engine guidance when setting or removing them. (developers.google.com)

If the curl response looks fine but Google still can’t fetch, collect the exact curl output (headers and full response) and the timestamps of attempted Googlebot requests found in your logs — those are the diagnostic items a host or a webmaster tool will need to trace rate limits, IP blocks, or server rules.

Recommended Answers

All 5 Replies

Member Avatar for JorgeM

While most submit as XML, the text file should work. Did you ensure that it is save as UTF-8 encoding?

Member Avatar for vivosmith

:D I did not think of that. Let me check if that works .

Member Avatar for vivosmith

So far it is not working, but I will try one more tweak.

Member Avatar for vivosmith

It said it could not fetch it now.

Edited by vivosmith
Member Avatar for Member #949455

It said it could not fetch it now.

I feel there's nothing wrong with your code.

Read this:

How can I check the X-Robots-Tag for a URL?

https://developers.google.com/webmasters/control-crawl-index/docs/faq#h31

Read this also:

Fetch as Google

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.