Member Avatar for David_329

I'm architecting a pipeline for scraping heavy media assets (images/video). Projected bandwidth is in the TB range/month.

I'm debating between:

Residential: Good anonymity, but expensive per GB.

Dedicated Servers: Fixed cost, but higher ban risk.

I'm testing Novada's unlimited dedicated servers (~$66/day for unmetered bandwidth) to handle the heavy lifting. The speed hits 1Gbps, which helps.

For those scraping heavy assets, do you mix proxy types? Or stick to dedicated hardware?

  • 2 Contributors
  • 1 Reply
  • 65 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by Dani
rproffitt commented: You didn't participate in your last post. This is spam. -4
Member Avatar for Dani

While DaniBot does scrape the web, it's not a core part of my business model so banning wouldn't be the absolute end of the world. I would figure out a way to pivot. However, as a general rule, I am very much against usage-based plans. For DaniWeb, specifically, we get a lot of bots scraping us, and the unpredictability of paying by the GB for DDoS attacks scares me. (Yes, we have 2 layers of DDoS protection, and neither works all that great. As you know, of course, there are lots of foolproof techniques to get around bot detection.)

I'm not that familiar with how this works, but can you pay a flat monthly rate for your own pool of IP addresses with your dedicated server?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.