dado.d 0 Newbie Poster

I was crawling (more specific: and it was working. I used PHP Symfony crawler, I even used proxies. I was testing it locally so I tried without proxies.

I added 5 seconds of sleep time between each requests, I tried random sleep between requests, I tried 10 seconds. After only few requests it gave me this page

I tried incognito and it gave me the same page. I guess it ties my crawling with IP address. How does it even recognizes those requests as non human?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of 1.18 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.