The Fastest way to read a .txt File - Page 2

[kux]

int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns

I hope you realize that the code in the last line above will likely cause buffer overflow. Lets say howmuchIread == READSIZE, which is the same as sizeof(buffer). Then buffer[howmuchIread] will be one byte beyond the end of the buffer.

if u look closer at the code, u will see that buffer is buffer[BUFFSIZE], and BUFFSIE = READSIZE + 1, so... no overrun there :)

[jencas]

If you are using MFC you can try something like:

CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

[kux]

If you are using MFC you can try something like:

CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

[jencas]

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

Yes, but loading the whole file in a single step makes it very fast. I agree, for files of a several 100 MB or more I wouldn't recommend this method, too.

[Dave Sinkula]

I am reading Comma delimited Large .txt files(About 50 Mb).

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.

[Jennifer84]

You are completely right about that. I have a large amout of code that I use after I red the values from the textfile. I mainly use the std:: namespace to substring, convert from text-number-text, like stringstream does, std::string::size_type, string1.length() etc...

I have experiment with this all day and found out that the System:: namespace have a much better performance than std:: namespace regarding the conversions and search in strings that I do. So I will have a great job to exchange all these operations and this will also improve performance and speed. However very nice.

As this example does this loop in 0.9 sec while the stringstream conversion that I use now will do this in 22 seconds.

double Number1;
	String^ Num = "4.34";

	for(int i = 0; i < 2000000; i++)
	{			
		Number1 = System::Convert::ToDouble(Num);
	}
	MessageBox::Show("Finish");

As I have discovered this better performance above with System:: I thinking of testing to also read a file with the System:: namespace. However I have never done that before and have a bit of a problem to find an example of how to do that.
I think I should use: System::IO:: StreamReader

if I would read the file that I have now with ifstream and getline as an example, it would look like this. How would this example look like with the System::IO namespace.
Still searching google for this.

std::string Text;
double n1, n2;
char Comma;

ifstream ReadFile("C:\\File.txt");

while( getline(ReadFile, Text, ',') )
{
          ReadFile >> n1;
          ReadFile >> Comma;
          ReadFile >> n2;
          ReadFile.get();
}

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.