I am working on one program that generates a lot of data...eventually it can be in GB range. And the way I am doing it now is just by writing to ascii file everything time... like this:
ofstream flowC("outputfile.out"); // declared at the start flowC << prevsid << ", " << threadid << endl; // inside working block - repeats many many times
However, this is not very efficient. First because I am writing ascii and because I am writing to dick every single time.
To improve this.. I want to write 2nd line above to a buffer of size 2048 or bigger. And then when buffer gets full I would write it to a file and emptry it. So I would write file in chunks of 2048 or so.... I also want to avoid writing file in ascii, but instead generate a binary file.
Can someone help me how to make this work correctly? I found some examples of how to write to a buffer but I am not sure how to write it from there to binary file.
char buffer; flowC.rdbuff()->pubsetbuf(buffer, 2048); ....... flowC << prevsid << ", " << threadid << endl;
I also eventually would like to compress the buffer before I write it to a file. I am not sure how this can be done yet in C or C++.
Edited 3 Years Ago by kdar