Member Avatar for nadiam

hi. i have a really basic question (really new to python). how do i read an entire binary file? from https://www.devdungeon.com/content/working-binary-data-python , data = binary_file.read() is to read the whole file at once. so i have a .dng file to test, which is 16.2mb, and i did:

with open("portrait1.dng", "rb") as binary_file:
        #Read the whole file at once
        data = binary_file.read()
        #Return the hexadecimal representation of the binary data, byte instance
        hexa = binascii.hexlify(data)
        with open("readprotrait1.raw", "wb") as write_file:
              write_file.write(hexa)

im using a hex editor to see if the data from .dng gets copied to the .raw file but only a portion of it gets copied. same result goes when i use a while loop like :

while True:
      data = binary_file.read()
      if not data:
           break
      #return hex rep and write

if i put a size(1024) in the .read(), only the last few lines of the data is written. so does it actually read the whole file and at the "write" part is where it gets complicated(doesn't write complete)? or am i missing something?

  • 2 Contributors
  • 2 Replies
  • 54K Views
  • 3 Hours Discussion Span
  • Latest Post Latest Post by Gribouillis

Dani AI

Generated

Two common pitfalls here: (1) calling read() with no size loads the entire file into RAM, which will thrash or freeze a machine on multi-GB inputs, and (2) binascii.hexlify() doubles the data size (2 ASCII chars per byte). If you are hexlifying just to "see" the data, expect a much larger output; it is not the same as the original raw bytes. To verify a full, correct read/copy, do not convert to hex; either compare sizes or compute a digest on the raw bytes.

  • Stream a copy without holding the file in memory:

    import shutil
with open("portrait1.dng", "rb") as src, open("copy.dng", "wb") as dst:
  shutil.copyfileobj(src, dst, length=1024*1024)  # 1 MiB chunks

  • Verify integrity without loading the file:

    import hashlib
h = hashlib.sha256()
with open("portrait1.dng","rb") as f:
  for chunk in iter(lambda: f.read(1024*1024), b""):
      h.update(chunk)
print(h.hexdigest())

If you must process in portions, keep your output file open once and append each processed chunk; repeatedly reopening with mode "wb" will overwrite earlier chunks and leave only the last block. For even lower overhead, consider readinto() with a preallocated bytearray or memoryview, or, for advanced use cases, memory mapping (mmap).

References: open(), Buffered I/O read semantics, binascii.hexlify, shutil.copyfileobj, hashlib.

Recommended Answers

All 2 Replies

Member Avatar for nadiam

Okay. I just found bytearray() which has accomplished my goals by:

with open("portrait1.dng", "rb") as binaryfile :
    myArr = bytearray(binaryfile.read())

with open("readfile.raw", "wb") as newFile:
    newFile.write(myArr)

but my questions still sort of stand because as i mentioned the current file i am testing is only 16.2mb and the real file which will be used is 8gb and i tested this code with that huge file and my laptop froze. so a couple of updated questions/thoughts:

  1. read and write obviously works but which operation uses most RAM? as one of it should be the reason why my laptop froze. is there a way to lessen that load? coz i can't keep testing and having my laptop freeze, that is surely unhealthy for it. my whole university life is on this (translation, if it dies i die).

  2. is there a better way to read the file? like portion by portion perhaps? how is that achieved? because i dont really need the write, the write is just for me to see that the file is wholly read. i need the read for other processes.
Member Avatar for Gribouillis

Read the file by chunks and also write chunks:

with open("portrait1.dng", "rb") as binaryfile :
    with open("readfile.raw", "wb") as newFile:
        while True:
            chunk = binaryfile.read(4096)
            if not chunk:
                break
            newFile.write(binascii.hexlify(chunk))

my whole university life is on this (translation, if it dies i die).

Make sure you backup your files regularly on a separate disk.

Edited by Gribouillis
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.