Member Avatar for phoenix06007

When a spider crawls your site, are they counted as traffic to your site?

  • 7 Contributors
  • 7 Replies
  • 130 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by moverspacker

Dani AI

Generated

Thread started by prompted quick replies from , , , , , and . The short answers in the thread miss the measurement nuance: what counts as "traffic" depends on the counting system used. Raw server logs record every HTTP request (including crawlers, monitoring services, and bad bots). Client-side analytics (Google Analytics, etc.) and webmaster tools (Search Console) apply different collection and filtering methods, so their numbers will diverge.

Server logs always show crawler requests. Common ways to spot them are user-agent strings, reverse DNS for known crawler IPs, and abnormal request patterns (very high request rate, single-page hits, or extremely short sessions). A quick log check example:

grep -iE 'bot|spider|crawl|bing|googlebot' access.log | less

Google provides guidance for verifying legit crawlers like Googlebot at Verifying Googlebot.

Client-side analytics rely on JavaScript beacons, so many bots never register there. Google Analytics offers known-bot filtering and view settings to reduce bot noise — see . Search Console’s crawl reports show how often major crawlers hit the site and reveal crawl errors. Combining logs + analytics + Search Console produces the most accurate picture of human versus crawler activity.

Practical checklist: enable bot filtering in analytics, run periodic server-log scans or use a log analyzer, consult Search Console for crawl stats, and only apply server-level blocks or rate limits after verifying offending IPs (robots.txt is advisory; blocking without verification risks blocking legitimate indexers).

Recommended Answers

All 7 Replies

Member Avatar for robseolas

Hi,

I don't think so, but if you want to know about your site traffic google analytics is good for you.

Member Avatar for freeonlinedatin

no i dont think so,its not comes under traffic.

Member Avatar for globalseo1

When spider craws the site it is not counted as traffic because crawling is the process of an engine requesting -- and successfully downloading -- a unique URL. Obstacles to crawling include no links to a URL, server downtime, robots exclusion, or using links (such as some JavaScript links) from which bots cannot find a valid URL.

Member Avatar for Denniz

Nope, I don't think spiders are considered as traffic.

Member Avatar for mackone

Yeah, I don't think they count as traffic.

Member Avatar for freeonlinedatin

hi,
no spiders doesnot count traffic

Member Avatar for moverspacker

intresting question

spider does not count the traffic of your website.

thanks

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.