DaniWeb Forum Index > Software Development > C++

The Fastest way to read a .txt File

Please support our C++ advertiser: Programming Forums - DaniWeb Sister Site

« First

•

Join Date: Jan 2008

Posts: 119

Reputation: kux is on a distinguished road

Solved Threads: 10

kux

Offline

Junior Poster

Re: The Fastest way to read a .txt File

#31

Sep 25th, 2008

•

Originally Posted by Ancient Dragon

Help with Code Tags

C++ Syntax (Toggle Plain Text)
	int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns
int howmuchIread = 0; while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) ) != 0 ) { buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns
I hope you realize that the code in the last line above will likely cause buffer overflow. Lets say howmuchIread == READSIZE, which is the same as sizeof(buffer). Then buffer[howmuchIread] will be one byte beyond the end of the buffer.

if u look closer at the code, u will see that buffer is buffer[BUFFSIZE], and BUFFSIE = READSIZE + 1, so... no overrun there

•

Join Date: Dec 2007

Posts: 360

Reputation: jencas is just really nice

Solved Threads: 69

jencas

Offline

Posting Whiz

Re: The Fastest way to read a .txt File

#32

Sep 25th, 2008

If you are using MFC you can try something like:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

If you are forced to reinvent the wheel at least try to invent a better one!

Please use code tags - Please mark solved threads as solved

•

Join Date: Jan 2008

Posts: 119

Reputation: kux is on a distinguished road

Solved Threads: 10

kux

Offline

Junior Poster

Re: The Fastest way to read a .txt File

#33

Sep 25th, 2008

•

Originally Posted by jencas

If you are using MFC you can try something like:
Help with Code Tags

C++ Syntax (Toggle Plain Text)
CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();
CFile inFile("test.txt", CFile::modeRead); CArchive archive(&inFile, CArchive::load, inFile.GetLength()); CString line; while (archive.ReadString(line)) { // do something with <line> here } archive.Close(); inFile.Close();

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

•

Join Date: Dec 2007

Posts: 360

Reputation: jencas is just really nice

Solved Threads: 69

jencas

Offline

Posting Whiz

Re: The Fastest way to read a .txt File

#34

Sep 25th, 2008

•

Originally Posted by kux

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

Yes, but loading the whole file in a single step makes it very fast. I agree, for files of a several 100 MB or more I wouldn't recommend this method, too.

If you are forced to reinvent the wheel at least try to invent a better one!

Please use code tags - Please mark solved threads as solved

•

Join Date: Apr 2004

Posts: 4,461

Reputation: Dave Sinkula has a brilliant future

Solved Threads: 254

Dave Sinkula Dave Sinkula is offline

Offline

long time no c

Re: The Fastest way to read a .txt File

#35

Sep 25th, 2008

•

Originally Posted by Jennifer84

I am reading Comma delimited Large .txt files(About 50 Mb).

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.

"One of the methods used by statists to destroy capitalism consists in establishing controls that tie a given industry hand and foot, making it unable to solve its problems, then declaring that freedom has failed and stronger controls are necessary." --Ayn Rand

•

Join Date: Feb 2008

Posts: 517

Reputation: Jennifer84 is an unknown quantity at this point

Solved Threads: 1

Jennifer84 Jennifer84 is offline

Offline

Posting Pro

Re: The Fastest way to read a .txt File

#36

Sep 25th, 2008

You are completely right about that. I have a large amout of code that I use after I red the values from the textfile. I mainly use the std:: namespace to substring, convert from text-number-text, like stringstream does, std::string::size_type, string1.length() etc...

I have experiment with this all day and found out that the System:: namespace have a much better performance than std:: namespace regarding the conversions and search in strings that I do. So I will have a great job to exchange all these operations and this will also improve performance and speed. However very nice.

As this example does this loop in 0.9 sec while the stringstream conversion that I use now will do this in 22 seconds.

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
	double Number1;
	String^ Num = "4.34";
 
	for(int i = 0; i < 2000000; i++)
	{			
		Number1 = System::Convert::ToDouble(Num);
	}
	MessageBox::Show("Finish");
	double Number1;
	String^ Num = "4.34";

	for(int i = 0; i < 2000000; i++)
	{			
		Number1 = System::Convert::ToDouble(Num);
	}
	MessageBox::Show("Finish");

As I have discovered this better performance above with System:: I thinking of testing to also read a file with the System:: namespace. However I have never done that before and have a bit of a problem to find an example of how to do that.
I think I should use: System::IO:: StreamReader

if I would read the file that I have now with ifstream and getline as an example, it would look like this. How would this example look like with the System::IO namespace.
Still searching google for this.

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
std::string Text;
double n1, n2;
char Comma;
 
ifstream ReadFile("C:\\File.txt");
 
while( getline(ReadFile, Text, ',') )
{
          ReadFile >> n1;
          ReadFile >> Comma;
          ReadFile >> n2;
          ReadFile.get();
}
std::string Text;
double n1, n2;
char Comma;

ifstream ReadFile("C:\\File.txt");

while( getline(ReadFile, Text, ',') )
{
          ReadFile >> n1;
          ReadFile >> Comma;
          ReadFile >> n2;
          ReadFile.get();
}

•

Originally Posted by Dave Sinkula

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.

Last edited by Jennifer84; Sep 25th, 2008 at 8:04 pm.