 |  |  |  |  |  |  |  |
Searching for text in huge (> 5GB) text files.
Please support our C++ advertiser: Intel Parallel Studio Home
 |
•
•
Join Date: Feb 2005
Posts: 466
Reputation: 
Solved Threads: 18
Posting Pro in Training
So here's what I'm trying to do. Basically, I have a huge text file of strings which are delimited by a delimiter. I would like to read each string separately based on the delimiter. Note that each chunk can span several lines, so I don't want to read it line by line. See this example:
Given that the file is 5GB, I can't read the entire contents of the file into memory. Is there a way do C (or C++) file handling with fseek or equivalent that will put out each chunk? I would like to do this as efficiently as possible, so I don't want to read each line at a time and concatenate the strings if I don't see the delimiter, etc. Is there an easy way?
(edit - unfortunately, this text editor won't properly show the fact that some of the text is on multiple lines, etc - hopefully the point was conveyed based on the wording)
C++ Syntax (Toggle Plain Text)
Given that the file is 5GB, I can't read the entire contents of the file into memory. Is there a way do C (or C++) file handling with fseek or equivalent that will put out each chunk? I would like to do this as efficiently as possible, so I don't want to read each line at a time and concatenate the strings if I don't see the delimiter, etc. Is there an easy way?
(edit - unfortunately, this text editor won't properly show the fact that some of the text is on multiple lines, etc - hopefully the point was conveyed based on the wording)
Last edited by winbatch; Jun 27th, 2006 at 6:42 pm.
•
•
Join Date: Feb 2002
Posts: 12,040
Reputation: 
Solved Threads: 128
I'm not really a c++ programmer but is there any reason why you aren't using a database? Perhaps a 5 gig file isn't the most efficient means of storing what you're trying to store - especially if you need the ability to do a linear search through it.
•
•
Join Date: Feb 2005
Posts: 466
Reputation:
Solved Threads: 18
Posting Pro in Training
•
•
•
•
Originally Posted by cscgal
I'm not really a c++ programmer but is there any reason why you aren't using a database? Perhaps a 5 gig file isn't the most efficient means of storing what you're trying to store - especially if you need the ability to do a linear search through it.
•
•
Join Date: Apr 2004
Posts: 4,361
Reputation:
Solved Threads: 241
Is one-character-at-a-time possible? What is the intended output?
"One of the methods used by statists to destroy capitalism consists in establishing controls that tie a given industry hand and foot, making it unable to solve its problems, then declaring that freedom has failed and stronger controls are necessary." --Ayn Rand
•
•
Join Date: Feb 2005
Posts: 466
Reputation:
Solved Threads: 18
Posting Pro in Training
•
•
•
•
Originally Posted by Dave Sinkula
Is one-character-at-a-time possible? What is the intended output?
In the end that's what I pretty much ended up doing. Just thought there might be a faster/more efficient approach.
use fopen and simple code a while loop not to end until the end of the file next get a pointer to the bigging of the file and read the first contents into the file now write that to a text file after that free the pointer and tell the program to skip one delimiter and the second time to skip two............
Another option is to read a good sized chunk of the file (5K, 1M, 10M) and operate on the data while it's in memory. When you have 10% of the chunk left unprocessed, move it to the beginning of the buffer and read the next chunk.
Reading a buffer at a time is much faster than a character at a time.
Reading a buffer at a time is much faster than a character at a time.
•
•
Join Date: Dec 2005
Posts: 5,850
Reputation:
Solved Threads: 749
http://www.cplusplus.com/ref/iostrea...m/getline.html
getline allows you to specify a delimiter of your choice, if it's a single char.
getline allows you to specify a delimiter of your choice, if it's a single char.
You can read the file after you open it using this method
your file pointer is advanced to the beginning of the next string
repeat the same procedure for the next string
you can put this in a subroutine or move the string contents somewhere else and reuse the same string again.
C++ Syntax (Toggle Plain Text)
your file pointer is advanced to the beginning of the next string
repeat the same procedure for the next string
you can put this in a subroutine or move the string contents somewhere else and reuse the same string again.
•
•
Join Date: Aug 2005
Posts: 15,406
Reputation:
Solved Threads: 1467
If the file is pretty much static (doesn't change very often) then you can create an index file that contains just the offsets into the master data file of the beginning of each record. Then when you want to read the 50th string just read the offset in the 50th record of the index file then seek to that position in the master data file. Each record in the index file is a 64-bit integer, so it is eash to seek to the desired record in the index file when you already know the record number.
|
Similar Threads
Other Threads in the C++ Forum
- Previous Thread: HELP!! don't understand c++ error messages
- Next Thread: Calling a var with other var value
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Related Forum Features
Latest C++ Posts
- Today's Posts
- Unanswered Threads
api array beginner binary bitmap c++ c/c++ calculator char char* class classes coding compile compiler console conversion convert count data database delete desktop developer directshow dll dynamic dynamiccharacterarray email encryption error file forms fstream function functions game getline google graph homeworkhelper iamthwee ifstream input int integer java lib linkedlist linker linux list loop looping loops map math matrix memory multiple news node number numbertoword output parameter pointer problem program programming project proxy python random read recursion recursive reference return rpg sorting string strings struct template templates test text tree unix url vector video visualstudio win32 windows winsock word wordfrequency wxwidgets
