[Python] Reducing proxy costs for 500k+ daily requests?

[David_329]

Hi everyone,

I'm optimizing a Python scraper (requests + multiprocessing) handling about 500k requests/day. The main bottleneck right now is bandwidth cost. Most residential pools are $8+/GB, which kills the margin.

I've refactored my rotation logic to be more efficient. Does anyone have recommendations for reliable residential pools under the $1/GB mark?

I'm currently testing Novada (found a Black Friday deal for $0.65/GB). Success rate is good so far with their rotation gateway. Any other budget-friendly alternatives for high volume?

[rproffitt]

This sounds like a classic XY Problem. Read https://en.wikipedia.org/wiki/XY_problem

[Dani]

I’m not sure what your use case is, but if it’s in your budget, have you considered renting datacenter space and renting your own IPs? I’m always about a fixed monthly price versus paying per GB.

[Dani]

Ugh. I had a whole post typed out on my phone and for some reason only the last paragraph posted.

What I was saying was that I wish I could be more help, but this is totally out of my world. DaniWeb has a couple of bots that we use for some functionality and for our SEO backlink tool, and our biggest problem with them are the constant 403 errors because we don’t use an ip pool. For our use case, it’s an acceptable loss compared to paying for the extra bandwidth.

I did a quick Google search and saw that everything online seems to be in the $1.50-$2 range, so if you’re at $1, you’re doing pretty well for yourself IMHO.

[rproffitt]

To me this sounds like the XY problem. You state or ask X but don't tell Y. Why are you doing this in the first place?

Then we can explore alternate solutions to the problem Y.

[Dani]

Why are you doing this in the first place?

They explained. They are using an army of bots to scrape the web to the tune of 500,000 HTTP requests per day. Why isn’t really relevant. However, this was asked as a digital marketing question, so it’s easy to make some guesses. Possibly a search engine. Possibly an AI tool. It doesn’t really matter. Either way, the answers given should be in that same vein.

Most firewalls have sophisticated techniques that block web scrapers. Additionally, bots are typically blocked via flood protection after repeated requests from the same IP address.

Therefore, what most people with scrapers do is they rent IP pools so that the bots make requests from all different high quality residential IP addresses. There are also techniques to make the HTTP requests appear non-automated, but that is beyond the scope of this question.

The OP states that the price he currently pays for these IP pools is cost prohibitive to his business model, and is asking if anyone else in this line of business (e.g. scraping) knows of any cheaper alternatives.

[rproffitt]

I see some want to focus on reduced costs vs examining what is being done along with the why.

Some business models don't work and the would be business person can't accept that.

So without the why part of the discussion you might be stuck in a forever loop of thinking "if only I could drive down costs." To wit, why do this if it's not a viable business model? Why would I need this data you collect? The XYPROBLEM is here. Address that.

[rproffitt]

While I do have ideas on how to reduce the 500k+ requests @David_329 hasn't engaged in the conversation beyond asking for help shopping for cheaper providers.

For example if this was some web scraper we could take a stab at how to reduce the frequency of scrapes. For example if the 500k requests were done daily then you test which scraped showed no change over a day and then push those off to a check every 2 day list. If the 2 day list finds some scrapes unchanged then push those off to a 3 day list. Repeat until you find a happy number. For example, depending on what I'm doing with the data (back to the Y question) I might set the longest scrape check to yearly.

Again, the Y (why) part of the discussion is missing so OP is again back to an XYProblem situation.