Member Avatar for SaaDwTk

I need an code to go this page:
http://steamcommunity.com/groups/KiFFERSTUEBCHEN/members

And grab the data beetwen:
http://steamcommunity.com/id/ and "

href="http://steamcommunity.com/id/otti">threeD</a><br />

so it will grab in this case the word "otti"
Theres a lot of lines like that, so i need a code that grab all the specified words of the page and store that on a litbox or some listtext, one word for line.

And when its done with all words of the page he need to goes to next page:
http://steamcommunity.com/groups/KiFFERSTUEBCHEN/members?p=2

so i need to make something like that

If grab on page is over then navigate to the next page "actualpage"+1

and if theres no specified word it stops.

Edited by SaaDwTk because: n/a
  • 3 Contributors
  • 3 Replies
  • 198 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by thomas49th

Dani AI

Generated

A clear, reliable approach that builds on ’s point about avoiding a WebBrowser: fetch each members page with an HTTP client and parse the HTML with an HTML parser (for example, HtmlAgilityPack). That avoids filling Temporary Internet Files, is faster for thousands of pages, and yields structured access to anchor hrefs so the short ID segment (the username-like token after the /id/ path) can be extracted reliably.

Core steps

  • Request the members page HTML with a modern HTTP client and a sensible User-Agent header.
  • Parse anchor nodes whose href contains /id/, read the href attribute, split on / and take the last segment, then strip any ? or # fragments.
  • Collect into a List(Of String) or a HashSet (for dedupe), then populate the ListBox on the UI thread (use Invoke/BeginInvoke for WinForms).
  • Advance pages by parsing pagination to find the total pages, or stop when a page returns zero matches or a non-success HTTP status.
  • Add polite throttling (Task.Delay), error handling, and respect robots.txt / site terms.

A concise VB.NET example (async, uses HtmlAgilityPack):

' Async example using HttpClient + HtmlAgilityPack
Imports System.Net.Http
Imports HtmlAgilityPack

Async Function FetchIdsAsync(groupMembersBase As String, startPage As Integer, maxPages As Integer) As Task(Of List(Of String))
    Dim results As New List(Of String)
    Using client As New HttpClient()
        client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (compatible; scraper/1.0)")
        For p = startPage To maxPages
            Dim url = String.Format("{0}?p={1}", groupMembersBase, p) ' base members URL + page number
            Dim html = Await client.GetStringAsync(url)
            Dim doc As New HtmlDocument()
            doc.LoadHtml(html)
            Dim nodes = doc.DocumentNode.SelectNodes("//a[contains(@href,'/id/')]")
            If nodes Is Nothing OrElse nodes.Count = 0 Then Exit For
            For Each n In nodes
                Dim href = n.GetAttributeValue("href","")
                If String.IsNullOrEmpty(href) Then Continue For
                Dim id = href.TrimEnd("/"c).Split("/"c).Last()
                Dim ix = id.IndexOfAny(New Char() {"?"c, "#"c})
                If ix >= 0 Then id = id.Substring(0, ix)
                If Not results.Contains(id) Then results.Add(id)
            Next
            Await Task.Delay(500) ' polite throttle
        Next
    End Using
    Return results
End Function

Troubleshooting and cautions: if SelectNodes returns nothing, the page structure may have changed or content is JS-rendered (then use a headless browser or the site API). If pages require login or return rate-limits, handle auth and backoff. Log page numbers that return empty results and test the routine on one page before running the full crawl. This directly addresses ’s extraction goal and clarifies the approach asked about by .

Recommended Answers

All 3 Replies

Member Avatar for codeorder

Since the pages you want to retrieve data from are in total of over 15 thousand, I would use Net.HttpWebResponse to retrieve the innerHtml of a web-page instead of using a WebBrowser.
This way you do not overload your Temporary Internet Files folder with Files that you have no use for.

Using Net.HttpWebResponse, you should be able to locate and extract the IDs and also extract the Total Pages from that first page.
Then just add +=1 and extract next page until you are at the Total number of pages available.

Edited by codeorder because: n/a
Member Avatar for SaaDwTk

Thank you very much codeorder.

Member Avatar for thomas49th

what is going on? How are you doing it?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.