Hi
Hope u r all well... here is my problem... i want to remove duplicate entries from a text file while maintaining the order of the text say if the text file contains

123zabc
456def
123abc
456def.
123def
456def
123abc
i want the result to be like
123zabc
456def
123abc
456def.
123def

i am currently using unix system calls to do this but its not working right every time...
i am using sort and unique function of unix...anyways
can some one please help me out with this... i do not mind getting the answer either in c or c++..
thanks

i thought of using hash function to remove duplicate entries but i am not sure whether that will work or not...

hehe....u free....can help me solve my problem??

You waited an entire 9 minutes before bumping your post? C'mon, we don't live on this board.

Read the entire file into an array of strings and loop through them.

the thing is the size of the file is in megabytes around 50 or 60 megabytes each.. i am keeping a record of all the files with in a system including system folders,hidden files e.t.c.... if i read the entire file in an array that would make the performance of the system slow...and i do not want to do that...anyhow i knew someone will say that i should have been clearer my fault my apologizes...

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

Let me put it this way -- leaving out important information makes it hard to give good help.

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

This certainly doesn't make anyone else here feel like helping. Sorry we're all such poor programmers. I have another option, but I guess you won't like it, either. He's all yours, AD...

thanks vijayan121 ...and so sorry waltp did not mean it like the way u took it...

This article has been dead for over six months. Start a new discussion instead.