Member Avatar for tendaishava

I would like to find out what this code snippet does. i know it involves extraction from a webpage format gb2312 but can ano\yone explain line by line, i appreciate the help... :)

Code blocks are created by indenting at least 4 spaces
... and can span multiple lines
 if (RgxNextPage.IsMatch(page))
        {


            nurl = RgxNextPage.Match(page).Value.Replace("pagepanel", string.Empty);
            Regex WebURL = new Regex(patternURL, RegexOptions.IgnoreCase);
            if (WebURL.IsMatch(nurl))
            {
                nurl = WebURL.Match(nurl).Value;
            }
            else
            {
                nurl = string.Empty;
            }


        }
        if (!nurl.Equals(url) && !string.IsNullOrEmpty(nurl))
        {
            string temp = readPage(nurl, "gb2312");
            ExtractResults(nurl, temp, list);
        }
  • 4 Contributors
  • 6 Replies
  • 256 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by thines01

Dani AI

Generated

Helpful, practical notes that build on , and while addressing the next steps described (merging three site crawls and removing duplicates).

The snippet should first cache Regex matches and validate the candidate URL before using it. That avoids repeated expensive matches and the null-reference problem mentioned by . Preferring a single Match, guarding with IsNullOrEmpty, and validating with Uri.TryCreate yields a much safer flow:

var nextMatch = RgxNextPage.Match(page);
if (nextMatch.Success)
{
    nurl = nextMatch.Value.Replace("pagepanel", string.Empty);
    var webMatch = WebURL.Match(nurl);
    nurl = webMatch.Success ? webMatch.Value : null;
}

if (!string.IsNullOrEmpty(nurl) && !string.Equals(nurl, url, StringComparison.OrdinalIgnoreCase))
{
    if (Uri.TryCreate(nurl, UriKind.Absolute, out var uri))
    {
        var temp = readPage(uri.ToString(), "gb2312");
        ExtractResults(uri.ToString(), temp, list);
    }
}

For combining results from three book sites: prefer ISBN as the canonical dedupe key. When ISBN is missing, normalize title+author (lowercase, strip punctuation, collapse whitespace) and use that as the dictionary key. If near-duplicates are suspected, apply a fuzzy match (Levenshtein or Jaro–Winkler) with a tuned threshold. Scores from each source should be normalized (e.g., score = 1.0 - (rank-1)/(n-1) or reciprocal rank) and merged by weighted average or max, then sort by the combined score. Example merging pattern:

string Normalize(string s) => Regex.Replace(s ?? "", @"\W+", " ").Trim().ToLowerInvariant();
// build map keyed by ISBN or Normalize(title+author), aggregate Score and Sources, then sort map.Values by Score.

Practical checklist: capture ISBN where possible, normalize fields at extraction time, log merged candidates for manual inspection, tune fuzzy thresholds on real data, and add unit tests for common edge cases. The combination of deterministic keys (ISBN/title+author) plus conservative fuzzy matching usually gives the best balance between removing duplicates and avoiding false merges.

Recommended Answers

All 6 Replies

Member Avatar for Mitja Bonca

Code is setting some variable "nurl" based on some matching - which is done by using Regular Expressions.
After that (in 2nd if statement) is checks the "nurl" varialbe and get some string from readPage() method.
Next (and last), is fires ExtractResults() method which obviously does some things (like setting or showing result; based on method`s name) - based on some parametes that were gathered and set previously in the code from above.

Hope it hepls.
bye

Member Avatar for Momerath

This is a poorly designed method. In line 3 it does a Regex match (IsMatch) then repeats the same match in the very next line (6). Regex is slow, do the matching once and save it. It does this again in lines 9 and 11.

Line 20 if nurl is null it will generate an exception as the test for null is after the use of the variable.

Member Avatar for tendaishava

thanxs alot guys

Member Avatar for thines01

A lot will depend on previous settings before the code gets to this point.
It could never reach this depending on other values (or crash as Momerath said).
Also, if patternURL is not properly formed, you'll get unexpected results.

Are you going to rewrite this function?

Member Avatar for tendaishava

yes i have done so. and tided up what needed to be. this is part of my project to build a system that will retrieve information from 3 different types of websites that are showing results from queries of books. in each website the program extracts all results and stores them in a text file. now that i have done that the next task is .

  1. combine all three programs form the different websites to one program (because each website is unique i had to do them differently) then have to sort results according to their rank.

  2. then remove duplicate results.

3.Lastly then present to the users from an interface that i have to design myself.

my main problem is what material can i go through to have knowledge on doing the result merging and sorting algorithm. en how to apply it to my code.

all types of advice really appreciated "positive en negative critisism all welcome"

Member Avatar for thines01

Result merging?
It sounds like you need a collection of items (either strings, key-value pairs or custom objects).
You can place the results together and evaluate them after they are collected.

Edited by thines01 because: typo
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.