Member Avatar for castajiz_2

Hy guys, haven t been here for a while,
If anyone can give me a idea or a way or a link on how to retrieve data from a website and store it in my application for further usage. The thing is that this data is being updated every month or so and when a update occurs on that website i again need this data updated in my program, so that my calculations concerning that data are always correct. My knowledge about websites are html,css, a litle bit of xml. Any advice would be greatly apreciated. I m using Visual studio 2012, C#.

  • 2 Contributors
  • 1 Reply
  • 218 Views
  • 5 Minutes Discussion Span
  • Latest Post Latest Post by Momerath

Dani AI

Generated

A pragmatic plan is more useful than a single library. The best route is an official, structured feed (API, JSON, XML, CSV, or RSS). If the site exposes one, consume that directly — it’s stable, fast, and legal. When no feed exists, parsing HTML is the fallback; HTML Agility Pack (as suggested) is a solid, tolerant parser for that job. If the page is rendered client-side (AJAX/JS), the browser Network tab often reveals the JSON/XHR endpoint that can be called directly instead of scraping HTML.

Suggested workflow

  1. Inspect the site with Developer Tools (Network tab) to find feeds or XHR endpoints.
  2. Check robots.txt and the site’s terms of use; prefer permission or an official feed.
  3. Fetch the data with HttpClient (.NET 4.5) or WebClient, then parse: XmlDocument/Json.NET for structured feeds or HtmlAgilityPack for HTML. Example pattern:
using System.Net.Http;
using HtmlAgilityPack;

async Task<string> DownloadHtml(string url)
{
    using(var client = new HttpClient())
        return await client.GetStringAsync(url);
}

async Task ParsePage(string url)
{
    var html = await DownloadHtml(url);
    var doc = new HtmlDocument();
    doc.LoadHtml(html);
    // select nodes, extract numbers, dates, etc.
}

Scheduling and storage
Run a scheduled job (Windows Task Scheduler or a small Windows Service) that checks the source monthly (or on a cadence matching updates). Use HTTP headers (ETag/Last-Modified) or a content hash to avoid unnecessary work. Store parsed results in a local DB (SQLite) or SQL Server and use an atomic update pattern (write to a temp table then swap) so calculations always see consistent data.

Cautions and hardening
Log parsing results and add unit tests that assert expected values/format. Watch for character encoding, regional number/date formats, and layout changes. Respect rate limits and legal terms; if parsing breaks frequently, request an official export/API from the data owner.

Member Avatar for Momerath

, examples and everything all there.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.