Fastest API to read/write files in C++

[madhusushmi]

I am looking for fastest API to read/write files. Can you please help?
I have checked such discussions here. But I am not able to figure out from those discussions. Please help.

[rubberman]

The C library read(), write() calls are probably the most efficient. What, exactly, are you trying to accomplish?

[nezachem]

Correction. read() and write() are POSIX compliant system calls; they do not belong to a C library; their presence is not covered by the Standard. Indeed, a Microsoft compiler doesn't provide them.

A compliant C library does provide fread() and fwrite() though.

[mike_2000_17]

>>But I am not able to figure out from those discussions.

That's probably because of two reasons that make this question a bit hard to answer. Before getting into it, of course, the two standard options that are available are C standard functions (i.e. fread() and fwrite(), or fscanf() and fprintf() for formatted IO), and the C++ standard functionalities (the <iostream> library). Otherwise, there are always OS-specific functions. The main difference between the two standard options is the amount of buffering done under-the-hood (buffering means that either data is pre-emptively read in blocks and cached before you actually demand it, or written data is accumulated before being actually written to the file in larger blocks). The C++ iostream library is designed with buffering mechanisms at its core, and as a result, implementations of it (provided by your compiler-vendor) generally have more sophisticated, multi-level buffering under-the-hood (and on top of any buffering mechanism that may be implemented by the OS). On the other hand, the C functionalities are generally just thin-wrappers for OS calls (so the only buffering done is that of the OS).

Buffering is the cause of the first complicating reason why there is no definitive and generic answer to your question. The amount of buffering needed for good performance and how the buffering is implemented depends a lot on the way that you read / write data. By that, I mean, in some applications you keep a file open and write small chunks of data once in a while (e.g. an event logger), in other applications you open a file and dump a massive amount of data into it (e.g. saving a raw image, like bitmaps), and then there is everything in between. It is not a trivial matter the figure out how much buffering (how many levels, how it is done, and the size of the buffers) is really optimal in any given situation. This is why, in some applications, the C functions (with less sophisticated buffering) work faster, while in other applications, the iostream functionalities work better.

The second reason why there is no generic answer to your question is simply that, by nature, I/O performance is highly dependent on the hardware, on the memory architecture, and the operating system.

Given these non-trivial factors, the file I/O demand (or pattern) (the first reason) and the system as a whole (the second reason), there is only one way to be sure what works better, that is, by experimenting. Try both methods (and any other OS-specific method too) and see which performs the best.

N.B. File I/O is often the bottleneck of many programs (and similarly for any other peripheral), that's because file I/O is slow. So, don't be surprised if it is slow. Generally, try to design your code such that you don't have to interact with files more often than you need to (which is, btw, what buffering really boils down to). A universal rule with computer architectures is that the further something is from the processor the slower it will be to interact with it. A good rule of thumb is the 100 rule, that is, access to registers is instantaneous (pretty much), the cache is about a 100 times slower to access, the main memory (RAM) is about 100 times slower than the cache, and the hard-drive is about a 100 times slower than the RAM (actually it is probably more in the ball-park of 1000 times slower). From some figures I've come across in the past, this rule is reasonable (one figure I can remember was along the lines of register - 2-3 clock cycles, cache - 200 clock cycles, RAM - 80,000 clock cycles, and HD - 100,000,000 clock cycles), but of course, the actual figures depend heavily on the system (and cache is now usually in multiple levels too).

[MasterHacker110]

Well you could use getline to read files and then this code to write to files:

ofstream myfile;
myfile.open("Hallo.txt")
myfile<<"Hallo"<<endl;
myfile.close();

to read from files:

string line;

ifsream file("FILE X.txt");

while(getline(file, line))
{
   // ohter functions to do wiht "line"
}

i know about hte read() and write() in c but if you are working on very big projects. i pays to use these that i gave.