Member Avatar for tcomc

These are the facts
for a real life case study.

unique ip address 160,000 this month
logged with a basic but accurate program
written in house
approx 8000 unique ip's a day .
227,000 hits this month
aw & in house agree .
+ 5000 zipped software downloads from site resources .

google analytics shows less than 10 visitors a day
real time analytics showing about this number
awstats
82 visitors a day
99.4 of all traffic is bookmarked/direct
reflecting the fact presumably that our freely available software integrates with a web site
unknown browsers/os in excess of 90% of all traffic
add to favorite 118% more than the number of visitors shown

can anyone throw some light on this puzzle

  • 3 Contributors
  • 6 Replies
  • 199 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by tcomc

Dani AI

Generated

Building on 's mystery and the points raised by and , here's a short, practical checklist to reconcile raw logs vs JavaScript analytics and to separate humans from automated clients.

Start by extracting the signal from the noise. Run a few quick audits against the raw access log (adapt field positions to your log format):

# unique IPs seen in the file
awk '{print $1}' access.log | sort -u | wc -l

# top IPs (who's hitting most)
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -40

# top requested URIs (what's being pulled)
awk -F\" '{print $2}' access.log | awk '{print $2}' | sort | uniq -c | sort -rn | head -40

# top user-agents (bots often show here)
awk -F\" '{print $6}' access.log | sort | uniq -c | sort -rn | head -40

# count downloads of zipped files
grep -i '\.zip' access.log | wc -l

# status code distribution
awk '{print $9}' access.log | sort | uniq -c | sort -rn

Use reverse DNS + forward-confirmation for suspect IPs before you call them "good" crawlers. Example:

host 1.2.3.4
dig -x 1.2.3.4 +short
host returned-name && host returned-name | grep 1.2.3.4  # forward-confirm

Also search the logs for requests to the tracking asset you actually serve (the analytics JS or pixel). If a large set of requests never fetch that asset, they won't show up in browser-based analytics.

Practical mitigations: serve large files through a download script that logs session/cookie presence (so automated fetches are obvious), add short-term rate limits or connection limits rather than blunt IP blocks, and tune AWStats/your log parser to use an up-to-date bot list. Be careful when blocking: verify crawlers via DNS checks first so you don't block legitimate search engines. If uncertainty remains after these checks, paste a sanitized sample of the top 50 lines and some head results of the commands above and the community can help interpret them.

Recommended Answers

All 6 Replies

Member Avatar for markchicobaby

Sure its not spammers/leechers? Set up the domain with leech protection in Apache.

Its probably not too hard for spammers to spoof IP addresses, or download via a bot net of IP addresses.

Are you sure those IPs are unique? Run them through sort -u or similar. Don't trust the tools, get the logs yourself and go through them.

Finally, have you examined the logs yourself, to see what the stats programs are seeing. You should be able to trace whether a user is accessing the site or just sucking down zips, in which case you've got bots coming at ya.

M

Member Avatar for Dani

Is your in house tracking software JavaScript based? I'd be willing to bet that it's counting bots and search engine spiders, which Google Analytics doesn't do. (Yes, they'd be many unique IPs).

Member Avatar for markchicobaby

Is your in house tracking software JavaScript based? I'd be willing to bet that it's counting bots and search engine spiders, which Google Analytics doesn't do. (Yes, they'd be many unique IPs).

Not that many. Would spiders really generate 8,000 unique IPs per day?

Member Avatar for Dani

As I am typing this, there are 636 Yahoo! slurp spiders, each with different IPs, crawling DaniWeb right this second. And I'm not even going to get into Googlebot!

Member Avatar for markchicobaby

What about the 5000 zip downloads? Spiders should be ignoring these?

Member Avatar for tcomc

Could be but
aw stats shows less than 5000 hits
can be attributed to spiders .
google shows next to no traffic and
no time spent on site .
Aw shows an average of 18 minutes 30 % 3000 visits in excess of 1 hour .
real time can not find these visits .
Raw log files 29 MB
showing lots of activity far more than aw
or google .

one set of stats is saying this site is healthy
another that it has tumbleweed blowing through it ! .

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.