> while( fread(mybigbuff, 1024, 1, fp) )
> {
> char* p = strtok(text,",");
You're not tokenising what you read.
You're not using the fread result to work out where the end of the buffer is
You're not appending a \0 to stop strtok() from going into the weeds (see previous posts)

This is a new way for me to read a file so I have to first understand the basic steps. I will assume a few things here to see If I understand the basics of this reading.

What happens first is that 1024 bytes is red into the buffer like a string. My question here. Is this the string: text ? If it is, the first step here is to tokenise this large string. I know that each line in the file contain 8 different values since 7 Commas delimits them. Since there is 1024 bytes there must be a lot of lines here, so I don´t understand how this will be tokenised. It must be a technique I dont know about. I think I just start out like this.

while( fread(mybigbuff, 1024, 1, fp) )
{
      char* p = strtok(text,",");

well, as salem's post said, u don't tokenize what u read.. the strtok should be
char* p = strtok(mybigbuff,",");
because mybigbuff is where u read from the file

now what I sugest u to do:

first decompose the problem: my ideea would be 4 u to first tokenize the buffer line by line, and then use one line however u want....thus simulating the fgets function

I have a text file tokenizer class which processes ~10 million tokens per second on low-medium CPU (it's the raw speed w/o file loading, ~0.2 sec for 50MB file). It loads the whole file in the core via fread but it's possible to map file or use another method. The file loading time is depended on the disk system configuration (and file fragmentation, of course). For my test file (50MB, 2.5 million tokens) loading time was ~2-3 seconds (~0.2 sec after the file was cashed in memory by OS).

If it seems interesting I can present sources...

#include <stdio.h>
#include <stdlib.h>

#include <iostream>
#include <string>
#include <vector>

using namespace std;


void tokenizestring( const std::string& src, const std::string& delim, vector<string>&tokens )
{
	string::size_type start = 0;
	string::size_type end;
        
        tokens.clear();
	for ( ; ; )
	{

		end = src.find ( delim, start );
		if ( end == string::npos )
			break;
		tokens.push_back ( src.substr ( start, end - start ) );

		start = end + delim.size();
	}

}

int main ( int argc, char* argv[] )
{

	FILE *fp;
	if ( (fp = fopen("c:\\test.txt", "rb")) == NULL )
	{
		printf( "unable to open file " );
		exit( 2 );
	}

	const int READSIZE = 1024;
	const int BUFFSIZE = READSIZE + 1;

	char buffer[BUFFSIZE]; // + 1 to have place for appending
				           //a terminal character

	vector<string> tokenized_byline;

	int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns
		string sBuffer = buffer;


		//the trick is I call a fgets to get the rest of the line and append
		//it to the rest of sBuffer
		//if the file position indicator is at a newline beginning, no problem,
		//we just read another line
		if ( !feof(fp))
		{
			fgets(buffer, READSIZE, fp);
			sBuffer += buffer ;
		}

		//tokenizestring tokenizes using /r/n  as delimiter, and stores the tokenized
		//strings to the vector tokenized_byline
		tokenizestring( sBuffer, "\r\n", tokenized_byline);
               
                //now u process the strings in tokenized_byline however u want

	}


	fclose( fp );


}

ok, u can get inspired from this

on my athlon 2Ghz it read a 50mb file in 3 secs

the algorithm works like this:

1. u read 1024 octets from file using fread
2. u store the result in a std::string
3. u then do a fgets to read the rest of the line ( most likley u end up with the file pos indicator inside a line, if not, there is no problem, u just read the next line to ). This won't slow u down... compered to the "just fgets" versions u saw before, the number of fgets calls will be at least 50 times smaller, depending on the linesizes
4. concatenete the rest of the line u just read with the rest of the string
5. u split the string by lines getting a std::vector
6. process the data in the tokenized_byline ( this is up 2 u: u will probably do another tokenization for each string in the vector using "," as separator to get u're required fields: the day and the other numbers u want , bla bla bla )

int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns

I hope you realize that the code in the last line above will likely cause buffer overflow. Lets say howmuchIread == READSIZE, which is the same as sizeof(buffer). Then buffer[howmuchIread] will be one byte beyond the end of the buffer.

Tokenize means breaking up your data into tokens based on delimiters.

For instance in your case

one,1,2,3,4
two,5,6,7,8.4
three,9,10,11,12

If your data were to be tokenized based on the "," and "\n" delimiters, you would have the tokens "one" "1" "2" "3" "4" and so forth.

In your case if you use fread to read a block of data, you basically have to process it until you reach the last valid token, and then tag on the rest of the buffer to the next buffer you read in.

For example if your token were a whole line, and your buffer had
one,1,2,3,4
two,5,6

Then you would process one,1,2,3,4 and preprend two,5,6 to the start of the next buffer. You should be careful about buffer overflows.

int howmuchIread = 0;
	while ( ( howmuchIread = fread ( buffer, 1, READSIZE, fp ) )  != 0 )
	{
		buffer[ howmuchIread ] = 0; //place terminal character to avoid overruns

I hope you realize that the code in the last line above will likely cause buffer overflow. Lets say howmuchIread == READSIZE, which is the same as sizeof(buffer). Then buffer[howmuchIread] will be one byte beyond the end of the buffer.

if u look closer at the code, u will see that buffer is buffer[BUFFSIZE], and BUFFSIE = READSIZE + 1, so... no overrun there :)

If you are using MFC you can try something like:

CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

If you are using MFC you can try something like:

CFile inFile("test.txt", CFile::modeRead);
CArchive archive(&inFile, CArchive::load, inFile.GetLength());
CString line;
while (archive.ReadString(line))
{
        // do something with <line> here
}
archive.Close();
inFile.Close();

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

hmmm,not shure, this kind of looks like you would load the entire file in memory. 4 very large file it would be very memory consuming i think

Yes, but loading the whole file in a single step makes it very fast. I agree, for files of a several 100 MB or more I wouldn't recommend this method, too.

I am reading Comma delimited Large .txt files(About 50 Mb).

And one would surmise that you are not simply copying the input to an output, that you are doing some sort of manipulations. This is probably very key to answering your question in full. Rather than micro-optimizing each particular function call, work on the overall algorithm. Or at least present an overview so that better answers for your overall effort may come as a result.