How do i read a file directly, not from the cache in python

[wiluo]

Hello,
I am trying to write a program in python to test the reading speed to a file, apparently,after the file is created, it's automatically in the cache and read() method will grab the data from the cache thus result in a much faster reading speed, how should i directly read from the file, not in the buffer or cache?

Thanks

[jcao219]

It's always read from the file. Where did you hear otherwise?

[griswolf]

It may be that wiluo is trying to use files for communication between processes, in which case it will be necessary to flush the write cache before trying to read the file inbound. Also conceivable: Multiple processes reading the same file. Most OSs cache some or all of the bytes in a file when they are first opened for reading. Not a Python issue (directly). Look here.

[jcao219]

It may be that wiluo is trying to use files for communication between processes, in which case it will be necessary to flush the write cache before trying to read the file inbound. Also conceivable: Multiple processes reading the same file. Most OSs cache some or all of the bytes in a file when they are first opened for reading. Not a Python issue (directly). Look here.

You mean flushing the I/O buffer, not the write cache. There is no cache.
Also very conceivable is he'll get lots of IOErrors with "access denied" from that, if a process has the file open, while another tries to write to it.

But I doubt the OP has that purpose,
instead I think he's trying to do this:

I am trying to write a program in python to test the reading speed to a file

[wiluo]

thanks guys, I am writing a program test read and write speed,i am first using write() to create a file with certain data and calculate the speed, then read from it, but I am guessing it will read from the cache, not from the data, so the read speed will be much faster then we expect. griswolf, do you know how could I clear the write cache?

Thanks a lot!

[jcao219]

There is no write cache.
There is no write cache.
It does not read from any cache.
It reads from the file.

[woooee]

Somebody should Google disk cache. This is the doc for flushing write commands.

file.flush()

Flush the internal buffer, like stdio‘s fflush(). This may be a no-op on some file-like objects.
Note: flush() does not necessarily write the file’s data to disk. Use flush() followed by os.fsync() to ensure this behavior.

A RAM cache used for reading would probably be OS specific and so you would have to turn it off via the OS( in Linux you can clear it by writing to proc/sys/vm/drop_caches) but I don't know how to turn it off. A cache is also part of the drive itself for relatively recent drives. This is a random choice just to see what size of a cache a modern drive has; this one is 32MB (500 Gig is only $59.99. Damn) http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=4914330&CatId=2459

[jcao219]

There is no cache...
is anybody listening?

Using open does not involve a cache at all!

There's only a cache when you use the
linecache module. Otherwise it's reading directly from the buffer.

[griswolf]

There is (very possibly) an OS-specific cache, and almost certainly a hardware cache that has nothing to do with Python...

Depending on what the OP is trying to do, the OS cache may or may not be relevant. I doubt you can do anything about the hard drive firmware.

[jcao219]

There is (very possibly) an OS-specific cache, and almost certainly a hardware cache that has nothing to do with Python...

Depending on what the OP is trying to do, the OS cache may or may not be relevant. I doubt you can do anything about the hard drive firmware.

Certainly not a hardware cache,
you see,
in Windows, you can call the CreateFile function in kernel32.dll with the flag FILE_FLAG_NO_BUFFERING, and that would a direct reading.
It's all about buffers.

This might also work: open("file.txt","r",0)#open with no buffer

[wiluo]

Thanks a lot guys, I am actually trying to rewrite a code that is written in cpp which uses the CreateFile function in kernel32.dll with the flag FILE_FLAG_NO_BUFFERING, I think open("file.txt","r",0)#open with no buffer is very helpful, that should does the same thing as that function right?
Thanks a lot.

[griswolf]

Whoa! C++ -> Python is not an obvious direction... but feel free ;) Now that I know what you are trying to do, I agree that the 0 buffering param is probably the right answer.

If the discussion is finished, don't forget to press the 'solved' button, which can be done only be the original poster.

[jcao219]

Thanks a lot guys, I am actually trying to rewrite a code that is written in cpp which uses the CreateFile function in kernel32.dll with the flag FILE_FLAG_NO_BUFFERING, I think open("file.txt","r",0)#open with no buffer is very helpful, that should does the same thing as that function right?
Thanks a lot.

Hopefully it's the same thing.
I really can't be sure.