removing duplicate lines from a text file while maintaining order of text

[bilalb1]

Hi
Hope u r all well... here is my problem... i want to remove duplicate entries from a text file while maintaining the order of the text say if the text file contains

123zabc
456def
123abc
456def.
123def
456def
123abc
i want the result to be like
123zabc
456def
123abc
456def.
123def

i am currently using unix system calls to do this but its not working right every time...
i am using sort and unique function of unix...anyways
can some one please help me out with this... i do not mind getting the answer either in c or c++..
thanks

[bilalb1]

i thought of using hash function to remove duplicate entries but i am not sure whether that will work or not...

[ricnyx]

hehe....u free....can help me solve my problem??

[WaltP]

hehe....u free....can help me solve my problem??

You waited an entire 9 minutes before bumping your post? C'mon, we don't live on this board.

Read the entire file into an array of strings and loop through them.

[bilalb1]

the thing is the size of the file is in megabytes around 50 or 60 megabytes each.. i am keeping a record of all the files with in a system including system folders,hidden files e.t.c.... if i read the entire file in an array that would make the performance of the system slow...and i do not want to do that...anyhow i knew someone will say that i should have been clearer my fault my apologizes...

[bilalb1]

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

[bilalb1]

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

[WaltP]

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

Let me put it this way -- leaving out important information makes it hard to give good help.

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

This certainly doesn't make anyone else here feel like helping. Sorry we're all such poor programmers. I have another option, but I guess you won't like it, either. He's all yours, AD...

[vijayan121]

you could use a std::set< std::string > ( or a map or a hashmap as you suggested ) to check for the duplicates. see this thread: http://www.daniweb.com/forums/thread81348.html

[bilalb1]

thanks vijayan121 ...and so sorry waltp did not mean it like the way u took it...