0

Hi
Hope u r all well... here is my problem... i want to remove duplicate entries from a text file while maintaining the order of the text say if the text file contains

123zabc
456def
123abc
456def.
123def
456def
123abc
i want the result to be like
123zabc
456def
123abc
456def.
123def

i am currently using unix system calls to do this but its not working right every time...
i am using sort and unique function of unix...anyways
can some one please help me out with this... i do not mind getting the answer either in c or c++..
thanks

4
Contributors
9
Replies
12
Views
9 Years
Discussion Span
Last Post by bilalb1
0

i thought of using hash function to remove duplicate entries but i am not sure whether that will work or not...

0

hehe....u free....can help me solve my problem??

You waited an entire 9 minutes before bumping your post? C'mon, we don't live on this board.

Read the entire file into an array of strings and loop through them.

0

the thing is the size of the file is in megabytes around 50 or 60 megabytes each.. i am keeping a record of all the files with in a system including system folders,hidden files e.t.c.... if i read the entire file in an array that would make the performance of the system slow...and i do not want to do that...anyhow i knew someone will say that i should have been clearer my fault my apologizes...

0

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

0

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

0

let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file :(

Let me put it this way -- leaving out important information makes it hard to give good help.

i wonder if ancient dragon is online ...he would have solved this in a minute....:(

This certainly doesn't make anyone else here feel like helping. Sorry we're all such poor programmers. I have another option, but I guess you won't like it, either. He's all yours, AD...

0

thanks vijayan121 ...and so sorry waltp did not mean it like the way u took it...

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.