I fixed it ^^

private void Form2_Load(object sender, EventArgs e)
        {
            openFileDialog1.Title = "Open Links List";
            openFileDialog1.Filter = "Text File|*.txt";
            openFileDialog1.FileName = "";
            openFileDialog1.ShowDialog();

            var file = openFileDialog1.FileName;

            if (file != "")
            {
                StreamReader read = new StreamReader(file);
                var items = read.ReadToEnd();

                string[] urls = items.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);

                var count = urls.Count();

                int i = 0;
                string[] item = new String[count];
                foreach (string u in urls)
                {
                    item[i] = u.Replace("\r", String.Empty);
                    i++;
                }

                read.Close();

                string[] new_urls = RemoveDups(item);

                StreamWriter write = new StreamWriter(file);

                foreach (string u in new_urls)
                {                    
                    write.WriteLine(u);
                }

                write.Close();

            }
        }

        public string[] RemoveDups(string[] s)
        {
            return s.Distinct().ToArray<String>();
        }

Hmm.. it works yes, but not for like 10k rows or so.. is there a way to remove duplicates from 100k+ items?

It should work for any number of items, but remember that the more items there are, the longer it takes to run the operation.

You will need to write your own method to do that. You can write an extension method and/or use regular expressions.

Yeah, that's a great idea, but have you any idea how I compare everything ?

So long as you know that the string will always be a URL, you should be able to get the Substring by finding the index of the 3rd forward slash.

So long as you know that the string will always be a URL, you should be able to get the Substring by finding the index of the 3rd forward slash.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.