Member Avatar for pen2satya

U can see some links on the first page of this site http://srimadbhagavatam.com/ and when you'll go one link among them you find another sub link.So i have to open all links to download the pages.I wonder the first link consists of 400 pages.So it's very uneasy and time taking to open all the links and then sub links and download.Is there any S/W or any way which can help me out ?

  • 3 Contributors
  • 2 Replies
  • 135 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by thewebhostingdi

Dani AI

Generated

A few practical options that solve the exact problem described by (and echoed by ): determine whether the site’s links are plain HTML or created dynamically with JavaScript, then pick one of two approaches.

If links are static HTML

  • Use an offline site-mirroring tool that follows hrefs and saves pages and resources. Two widely used choices are HTTrack (GUI + filters) and GNU Wget (command-line).
  • Example wget command (replace the placeholder URL and test on a small subtree first):
    wget --mirror --convert-links --page-requisites --adjust-extension --wait=1 --limit-rate=200k -P ./saved_site http://target-site/

    Options explained: mirror for recursive download, convert-links to make local browsing work, page-requisites to pull CSS/images, wait/limit-rate to avoid hammering the server, and a local output directory.

If links are generated by JavaScript or require interaction

  • Simple mirroring tools won’t see dynamically created anchors. A headless browser approach (Puppeteer or Selenium) is appropriate: load pages, wait for the DOM/network to settle, extract anchor hrefs, queue new pages up to the desired depth, and save the rendered HTML/resources. This requires a small script but handles AJAX-driven sites.

Practical tips and cautions

  • Look first for a sitemap.xml or an index page; that can provide a URL list without crawling.
  • Use include/exclude filters or a depth limit to keep the job to the relevant 400 pages and avoid crawling the whole domain.
  • Respect robots.txt and site terms; throttle requests and test on a few pages before a full run. If the site is large or traffic-sensitive, request permission from the owner.
  • As noted, a dedicated downloader is the right idea; choose the tool based on whether the site needs JavaScript execution, authentication, or careful throttling.
Member Avatar for ayyam

dear .....i face the same difficulties ..so waits ..to see the result

Member Avatar for thewebhostingdi

WebsiteCopier is the software that you need. There may also be some other software available for this purchase.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.