Ad:

C++ Discussion Thread
Unsolved
Views: 14379

You are currently viewing page 4 of this multi-page discussion thread; Jump to the first page

Sep 25th, 2008

Re: The Fastest way to read a .txt File

Click to Expand / Collapse Quote originally posted by Ancient Dragon ...

C++ Syntax (Toggle Plain Text)
	int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns
int howmuchIread = 0; while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) ) != 0 ) { buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns
I hope you realize that the code in the last line above will likely cause buffer overflow. Lets say howmuchIread == READSIZE, which is the same as sizeof(buffer). Then buffer[howmuchIread] will be one byte beyond the end of the buffer.

if u look closer at the code, u will see that buffer is buffer[BUFFSIZE], and BUFFSIE = READSIZE + 1, so... no overrun there

kux

Reputation Points: 66

Solved Threads: 11

Junior Poster

Offline

119 posts
since Jan 2008

Permalink

Sep 25th, 2008

Re: The Fastest way to read a .txt File

If you are using MFC you can try something like:

C++ Syntax (Toggle Plain Text)
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

jencas

Reputation Points: 395

Solved Threads: 71

Posting Whiz

Offline

362 posts
since Dec 2007

Permalink

Sep 25th, 2008

Re: The Fastest way to read a .txt File

Click to Expand / Collapse Quote originally posted by jencas ...

If you are using MFC you can try something like:
C++ Syntax (Toggle Plain Text)
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();
CFile inFile("test.txt", CFile::modeRead); CArchive archive(&inFile, CArchive::load, inFile.GetLength()); CString line; while (archive.ReadString(line)) { // do something with <line> here } archive.Close(); inFile.Close();

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

kux

Reputation Points: 66

Solved Threads: 11

Junior Poster

Offline

119 posts
since Jan 2008

Permalink

Sep 25th, 2008

Re: The Fastest way to read a .txt File

Click to Expand / Collapse Quote originally posted by kux ...

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

Yes, but loading the whole file in a single step makes it very fast. I agree, for files of a several 100 MB or more I wouldn't recommend this method, too.

jencas

Reputation Points: 395

Solved Threads: 71

Posting Whiz

Offline

362 posts
since Dec 2007

Permalink

Sep 25th, 2008

Re: The Fastest way to read a .txt File

Click to Expand / Collapse Quote originally posted by Jennifer84 ...

I am reading Comma delimited Large .txt files(About 50 Mb).

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.

Dave Sinkula

Team Colleague

Reputation Points: 2780

Solved Threads: 312

long time no c

Offline

4,790 posts
since Apr 2004

Permalink

Sep 25th, 2008

Re: The Fastest way to read a .txt File

You are completely right about that. I have a large amout of code that I use after I red the values from the textfile. I mainly use the std:: namespace to substring, convert from text-number-text, like stringstream does, std::string::size_type, string1.length() etc...

I have experiment with this all day and found out that the System:: namespace have a much better performance than std:: namespace regarding the conversions and search in strings that I do. So I will have a great job to exchange all these operations and this will also improve performance and speed. However very nice.

As this example does this loop in 0.9 sec while the stringstream conversion that I use now will do this in 22 seconds.

C++ Syntax (Toggle Plain Text)
	double Number1;
	String^ Num = "4.34";
 
	for(int i = 0; i < 2000000; i++)
	{			
		Number1 = System::Convert::ToDouble(Num);
	}
	MessageBox::Show("Finish");
	double Number1;
	String^ Num = "4.34";

	for(int i = 0; i < 2000000; i++)
	{			
		Number1 = System::Convert::ToDouble(Num);
	}
	MessageBox::Show("Finish");

As I have discovered this better performance above with System:: I thinking of testing to also read a file with the System:: namespace. However I have never done that before and have a bit of a problem to find an example of how to do that.
I think I should use: System::IO:: StreamReader

if I would read the file that I have now with ifstream and getline as an example, it would look like this. How would this example look like with the System::IO namespace.
Still searching google for this.

C++ Syntax (Toggle Plain Text)
std::string Text;
double n1, n2;
char Comma;
 
ifstream ReadFile("C:\\File.txt");
 
while( getline(ReadFile, Text, ',') )
{
          ReadFile >> n1;
          ReadFile >> Comma;
          ReadFile >> n2;
          ReadFile.get();
}
std::string Text;
double n1, n2;
char Comma;

ifstream ReadFile("C:\\File.txt");

while( getline(ReadFile, Text, ',') )
{
          ReadFile >> n1;
          ReadFile >> Comma;
          ReadFile >> n2;
          ReadFile.get();
}

Click to Expand / Collapse Quote originally posted by Dave Sinkula ...

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.

Last edited by Jennifer84; Sep 25th, 2008 at 8:04 pm.

Jennifer84

Reputation Points: 10

Solved Threads: 1

Posting Pro

Offline

563 posts
since Feb 2008

Previous
1
2
3
Page 4

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in C++ Forum Timeline: Compilation error

Next Thread in C++ Forum Timeline: system call to read the windows kernel message queue

DaniWeb Message

Cancel Changes

User Name
Password