How to remove duplicates from array

[JannuBl22t]

Hey!

I have a problem.. I need to remove items from array that are duplicated here's my code:

private void Form2_Load(object sender, EventArgs e)
        {
            openFileDialog1.Title = "Open Links List";
            openFileDialog1.Filter = "Text File|*.txt";
            openFileDialog1.FileName = "";
            openFileDialog1.ShowDialog();

            var file = openFileDialog1.FileName;

            if (file != "")
            {
                StreamReader read = new StreamReader(file);
                var items = read.ReadToEnd();
                string[] urls = items.Split('\n');
                var url = RemoveDups(urls);

                read.Close();

                //StreamWriter write = new StreamWriter(file);

                foreach (string i in urls)
                {
                    //write.WriteLine(i);
                    MessageBox.Show(i);
                }

                //write.Close();

                /*StreamWriter writer = new StreamWriter(file);

                string currentLine;
                HashSet<string> previousLines = new HashSet<string>();

                while ((currentLine = read.ReadLine()) != null)
                {
                    if (previousLines.Add(currentLine))
                    {
                        writer.WriteLine(currentLine);
                    }
                }*/
            }
        }

        public string[] RemoveDups(string[] s)
        {
            var count = s.Count();

            string[] arr = new String[count];

            for (int i = 0; i < count; i++)
            {
                if (!arr.Contains(s[i]))
                {
                    arr[i] = s[i];
                }
            }

            return arr;

        }

Somehow it doesnt work

[ddanbe]

Try this:

class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("***original array***");
            string[] strs = new string[4] { "one", "two", "two", "three" };
            for (int i = 0; i < strs.Length; i++)
            {
                Console.WriteLine(strs[i]);
            }
            Console.WriteLine("***distinct array***");
            var Results = strs.Distinct();
            foreach (var item in Results)
            {
                Console.WriteLine(item);
            }
            Console.ReadLine();
        }
    }

[JannuBl22t]

But what's that distinct array ?

[ddanbe]

Distinct is not an array, it is a method.
It puts all the distinct string values of the strs array into the Results array.
Arrays in C# are alwas with [] never with () as in VB.

[JannuBl22t]

Hm, I tried this:

public string[] RemoveDups(string[] s)
        {

            var new_list = s.Distinct();

            return new_list.ToArray();
        }

But it doesnt want to work..

[Ketsuekiame]

Please give an example of your input and output.

[JannuBl22t]

Well.. it is like a duplicate removal script so you open it it automatically shows you a "open file" dialog.. and then you select a text file that looks like:

http://google.com
http://daniweb.com
http://google.com
http://webhostingtalkforums.com

and then this script will automatically remove in this example http://google.com and saves all unique domains to the same file where it took them.

[Ketsuekiame]

When you use those as input, what do you get back out after you use Distinct on the array?

[JannuBl22t]

Everything is the same.. nothing has been removed or added.

[Ketsuekiame]

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace DaniWebTest
{
    class Program
    {
        static void Main(string[] args)
        {
            String[] myString = new String[] { "http://google.com", "http://daniweb.com", "http://google.com", "http://webhostingtalkforums.com" };
            IEnumerable<String> disctinctUrl = myString.Distinct();
            foreach (String theString in disctinctUrl)
            {
                Console.WriteLine(theString);
            }
            Console.ReadLine();
        }

    }
}

I just ran the above code and it worked as predicted. IT only printed three URL's, all unique. Converting to an array won't affect the contents so much as to add entries that shouldn't be there.

Please try running that above code and see what happens.

[JannuBl22t]

I modified it to:

public string[] RemoveDups(string[] s)
        {
            IEnumerable<string> rmv = s.Distinct();
            var count = s.Count();
            string[] urls = new String[count];

            int n = 0;
            foreach (string i in rmv)
            {
                urls[n] = i;
                n++;
            }

            return urls;
        }

But it still don't want to work. :/

[Ketsuekiame]

try return s.Distinct().ToArray<String>();

[JannuBl22t]

Sorry it doesn't work.

Here's how I view them:

string[] new_urls = RemoveDups(urls);

                foreach (string u in new_urls)
                {
                    MessageBox.Show(u);
                }

[Ketsuekiame]

If it isn't working, there is something wrong with your input, they must not be equal. Check *exactly* what is going in. The Distinct method *does* work and I've shown you code which works.

The only thing remaining is the fact that the input is different where you expect it to be the same.

[JannuBl22t]

My code looks like this:

private void Form2_Load(object sender, EventArgs e)
        {
            openFileDialog1.Title = "Open Links List";
            openFileDialog1.Filter = "Text File|*.txt";
            openFileDialog1.FileName = "";
            openFileDialog1.ShowDialog();

            var file = openFileDialog1.FileName;

            if (file != "")
            {
                StreamReader read = new StreamReader(file);
                var items = read.ReadToEnd();

                string[] urls = items.Split('\n');
                string[] new_urls = RemoveDups(urls);

                foreach (string u in new_urls)
                {
                    MessageBox.Show(u);
                }


            }
        }

        public string[] RemoveDups(string[] s)
        {
            return s.Distinct().ToArray<String>();
        }
    }

And that text file I'm opening looks like this:

http://www.webmastertalkforums.com/members/jaan.html
http://webhostingtalkforums.com
http://google.com
http://google.com

[Ketsuekiame]

Ah, I was correct ;)

Your input is being split by '\n'. However, this leaves '\r' on the end of each line except the last line.

So if you breakpoint and look at your urls variable. You will see something like:

http://www.webmastertalkforums.com/members/jaan.html\r
http://webhostingtalkforums.com\r
http://google.com\r
http://google.com

So in computer terms, the last google.com is different to the google.com above it.

I recommend one of the following:
1. Split by "\r\n". This won't work for all files.
2. Split by '\n', replace any '\r' with String.Empty
3. Read each line in separately and remove any instance of '\r' and '\n'

That will solve your problem

[JannuBl22t]

Hmmm.. I tried like this but it fails:

private void Form2_Load(object sender, EventArgs e)
        {
            openFileDialog1.Title = "Open Links List";
            openFileDialog1.Filter = "Text File|*.txt";
            openFileDialog1.FileName = "";
            openFileDialog1.ShowDialog();

            var file = openFileDialog1.FileName;

            if (file != "")
            {
                StreamReader read = new StreamReader(file);
                var items = read.ReadToEnd();

                string[] urls = items.Split('\n');

                var count = urls.Count();

                string[] ur = new String[count];

                int num = 0;
                foreach(string u in urls) 
                {
                    ur[num] = u.Replace('\r', ' ');
                }

                string[] new_urls = RemoveDups(ur);

                foreach (string u in new_urls)
                {
                    MessageBox.Show(u);
                }


            }
        }

        public string[] RemoveDups(string[] s)
        {
            var count = s.Count();

            int i = 0;
            string[] items = new String[count];

            foreach (string u in s)
            {
                try
                {
                    items[i] = u.Replace('\r', ' ');
                }
                catch (Exception e)
                {
                    items[i] = "Failed" + e.ToString();
                }
            }

            return items.Distinct().ToArray<String>();
        }

[Ketsuekiame]

But now you're replacing '\r' with a space character, which is still different to no character at all. Please read what I put again :P

[JannuBl22t]

I dont't know how to remove \r :/

I tried:

string[] urls = items.Split('\r\n');

But it fails..

[Ketsuekiame]

You need to replace '\r' with String.Empty

'\r\n' fails because '' indicates a single character \r is one character, \n is another character. You would need to use "" to indicate a string (multiple characters) and String.Empty is part of the String class.

[JannuBl22t]

But Replace wants char not string.. that's another error i get.. :/

[Ketsuekiame]

Use speech marks instead of apostrophes. So myString.Replace("\r", String.Empty);

[JannuBl22t]

Hmmm.. it works :D thank you, but now it shows "http://google.com" and a empty messagebox, it doesnt show an array without duplicates. :/

[Ketsuekiame]

Where you split the files, change this to items.Split('\n', StringSplitOptions.RemoveEmptyEntries); as you may have a blank line at the end of your file. (Or in between any other line)

[JannuBl22t]

It doesnt work.. :/ it gives me some kind of error again:

http://i55.tinypic.com/bi07mf.jpg

[Ketsuekiame]

I was hoping you'd figure it out yourself... ;) string[] urls = items.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries); is the actual line you need

[JannuBl22t]

I'm sorry, I'm not very pro @ C#. Anyway.. I still see empty messagebox and then http://google.com

[Ketsuekiame]

Use breakpoints to see what your array contains.

[jeffreylee]

As far as removing a duplicate array is concerned we need to use the array properties which will do the needful

[Mitja Bonca]

Try with using HasSet collection:

class Program
    {
        static void Main(string[] args)
        {
            string[] array = { "a", "b", "c", "b", "d", "e", "c" };
            string[] result = RemovingDuplicates(array);
        }

        private static string[] RemovingDuplicates(string[] array)
        {
            HashSet<string> set = new HashSet<string>(array);
            string[] result = new string[set.Count];
            set.CopyTo(result);
            return result;
        }
    }