C++ RSS C++ RSS

removing duplicate lines from a text file while maintaining order of text

Please support our C++ advertiser: Intel Parallel Studio Home
Reply

•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

removing duplicate lines from a text file while maintaining order of text

 
0
  #1
Nov 25th, 2007
Hi
Hope u r all well... here is my problem... i want to remove duplicate entries from a text file while maintaining the order of the text say if the text file contains

123zabc
456def
123abc
456def.
123def
456def
123abc
i want the result to be like
123zabc
456def
123abc
456def.
123def

i am currently using unix system calls to do this but its not working right every time...
i am using sort and unique function of unix...anyways
can some one please help me out with this... i do not mind getting the answer either in c or c++..
thanks
Reply With Quote Quick reply to this message  
•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #2
Nov 25th, 2007
i thought of using hash function to remove duplicate entries but i am not sure whether that will work or not...
Reply With Quote Quick reply to this message  
•
•
Join Date: Nov 2007
Posts: 21
Reputation: ricnyx is an unknown quantity at this point 
Solved Threads: 0
ricnyx ricnyx is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #3
Nov 25th, 2007
hehe....u free....can help me solve my problem??
Reply With Quote Quick reply to this message  
•
•
Join Date: May 2006
Posts: 3,114
Reputation: WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of 
Solved Threads: 281
Moderator
WaltP's Avatar
WaltP WaltP is offline Offline
Posting Sensei

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #4
Nov 25th, 2007
•
•
•
•
Originally Posted by ricnyx View Post
hehe....u free....can help me solve my problem??
You waited an entire 9 minutes before bumping your post? C'mon, we don't live on this board.

Read the entire file into an array of strings and loop through them.
The 3 Laws of the Procrastination Society:
1) Never do today that which can be put off until tomorrow
2) Tomorrow never comes
Reply With Quote Quick reply to this message  
•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #5
Nov 25th, 2007
the thing is the size of the file is in megabytes around 50 or 60 megabytes each.. i am keeping a record of all the files with in a system including system folders,hidden files e.t.c.... if i read the entire file in an array that would make the performance of the system slow...and i do not want to do that...anyhow i knew someone will say that i should have been clearer my fault my apologizes...
Reply With Quote Quick reply to this message  
•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #6
Nov 25th, 2007
let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file
Reply With Quote Quick reply to this message  
•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #7
Nov 25th, 2007
i wonder if ancient dragon is online ...he would have solved this in a minute....
Reply With Quote Quick reply to this message  
•
•
Join Date: May 2006
Posts: 3,114
Reputation: WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of WaltP has much to be proud of 
Solved Threads: 281
Moderator
WaltP's Avatar
WaltP WaltP is offline Offline
Posting Sensei

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #8
Nov 25th, 2007
•
•
•
•
Originally Posted by bilalb1 View Post
let me put it this way about 17000 lines in the file take up to 1 to 1.5 megabytes so if i do as u suggested...even in this case i would need around 17000 * 1000 matches in most cases... and 17000*17000 matches in worst case only for 1/50 th of the file
Let me put it this way -- leaving out important information makes it hard to give good help.


•
•
•
•
Originally Posted by bilalb1 View Post
i wonder if ancient dragon is online ...he would have solved this in a minute....
This certainly doesn't make anyone else here feel like helping. Sorry we're all such poor programmers. I have another option, but I guess you won't like it, either. He's all yours, AD...
The 3 Laws of the Procrastination Society:
1) Never do today that which can be put off until tomorrow
2) Tomorrow never comes
Reply With Quote Quick reply to this message  
•
•
Join Date: Dec 2006
Posts: 1,089
Reputation: vijayan121 is a name known to all vijayan121 is a name known to all vijayan121 is a name known to all vijayan121 is a name known to all vijayan121 is a name known to all vijayan121 is a name known to all 
Solved Threads: 164
vijayan121 vijayan121 is offline Offline
Veteran Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #9
Nov 25th, 2007
you could use a std::set< std::string > ( or a map or a hashmap as you suggested ) to check for the duplicates. see this thread: http://www.daniweb.com/forums/thread81348.html
Reply With Quote Quick reply to this message  
•
•
Join Date: Aug 2007
Posts: 18
Reputation: bilalb1 is an unknown quantity at this point 
Solved Threads: 1
bilalb1 bilalb1 is offline Offline
Newbie Poster

Re: removing duplicate lines from a text file while maintaining order of text

 
0
  #10
Nov 26th, 2007
thanks vijayan121 ...and so sorry waltp did not mean it like the way u took it...
Reply With Quote Quick reply to this message  
Reply

Quick Reply to Thread
This thread is more than three months old.
Perhaps start a new thread instead?
•
•
•
•
•
•
•
•
Message:
•
•
•
•



Similar Threads
Other Threads in the C++ Forum
Thread Tools Search this Thread



api array based beginner binary bitmap c++ c/c++ calculator char char* class classes coding compile compiler console conversion count data database delete deploy desktop developer directshow dll dynamiccharacterarray email encryption error file forms fstream function functions game getline google graph homeworkhelp homeworkhelper iamthwee ifstream input int integer lib linkedlist linux list loop looping loops map math matrix memory multiple news node number numbertoword output parameter pointer problem program programming project proxy python random read recursion recursive reference return rpg sorting string strings struct template templates test text tree unix url vector video visual visualstudio win32 windows winsock word wordfrequency wxwidgets
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC