Member Avatar for mairtin.obrian

Hello
Does anyone know of some good Web Site Scraping software that they can reccomend
Thanks

  • 4 Contributors
  • 3 Replies
  • 203 Views
  • 2 Years Discussion Span
  • Latest Post Latest Post by RonVelder

Dani AI

Generated

Short, practical follow‑up for : the “best” scraper depends on three things — the site you need, the volume, and whether the site exposes an API. Public APIs (use them) avoid the headache of parsing and legal risk; if you must scrape, always check the site’s robots.txt and the site’s terms before you start. (rfc-editor.org)

Legal and ethical note: US case law (for example, the hiQ v. LinkedIn decisions) shows outcomes can vary and courts have treated scraping of publicly accessible data differently than accessing behind barriers. That means scraping public pages is not an automatic green light for commercial projects — review terms and get legal advice for anything high‑volume or revenue‑critical. (law.justia.com)

Practical workflow and tool guidance: for quick one‑offs use simple HTTP requests + an HTML parser (requests + BeautifulSoup in Python); for robust, repeatable crawls use a crawler framework that supports politeness, retries and auto‑throttling (Scrapy is a common choice); for JS/AJAX pages use a real browser automation tool (Playwright or Puppeteer) so you render scripts instead of reverse‑engineering every XHR. Pick tooling that fits your deployment needs (if you need distributable EXEs, prefer languages/tools that package cleanly). (crummy.com)

Quick operational tips: inspect the site’s network/XHR calls first — many sites have undocumented JSON endpoints that are easier to use; add conservative rate limits and exponential backoff; log and monitor failures so you can detect blocks; plan for proxy rotation and CAPTCHA handling only when necessary; and favor licensed/paid access if you need commercial‑scale reliability. Be mindful that many sites now use bot management, so defensive measures exist on the server side. (developers.cloudflare.com)

If you want, provide the target site (domain) and the data you need and the expected volume — that lets you get a specific, minimal workflow recommendation.

Recommended Answers

All 3 Replies

Member Avatar for Ancient Dragon

I don't know a thing about it, but did you check google?

Member Avatar for jthralow

Hello
Does anyone know of some good Web Site Scraping software that they can reccomend
Thanks

I am a marketing consultant for a company that was trying to scrape several sites to get their pricing competitive with everyone else online. I spent a good month learning the programs out there and using them until I found one that actually worked. Not only did it work it blew me way. These guys have a great vision making the system open enough to have the data harvested and then sent back to the user via REST, FTP, or email. The client loves me as the closest competitor was $50,000/year and they did not work as well. Anyway try mozenda.com for your web scraper I think you will be very happy. I wish I had read this before spending so much time trying the other ones.

In full disclosure I have to say that I Invested in this company (Mozenda Inc.) after trying their tool.

Hope this helps,
Jon

Member Avatar for RonVelder

I use Djuggler for web scraping. Works perfectly for me so far, including sites that use AJAX. Best of all you can compile the scrape scripts as standalone executables.

R.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.