Member Avatar for Banjoplucker

Hello,

I'm new here, so please be gentle. I have posted my code at the following page:

In short, I need to read in my file, block by block, but have the re.finditer regular expression report the offsets relative to the beginning of the overall file, rather than relative to each freas 4096 byte block as it is read in.

Thanks for your help

Banjoplucker

  • 2 Contributors
  • 2 Replies
  • 123 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by Banjoplucker

Dani AI

Generated

A simple, robust plan builds on what pointed out: keep a running file-position and also handle matches that straddle the boundary between two blocks. Two practical patterns that work well for large files follow.

A low-dependency approach is to read fixed-size blocks but keep a short tail from the previous block so a match cannot be split across reads. Choose overlap equal to the maximum possible match length minus one (or a safe conservative value). The buffer you search is tail + block; compute the global file offset by adjusting the match start by the amount you prefixed from the previous block.

import re

pattern = re.compile(b'yourpattern')   # bytes pattern for binary files
block_size = 4096
overlap = 128  # set based on expected max match length

offset = 0
tail = b''

with open('bigfile.bin', 'rb') as f:
    while True:
        chunk = f.read(block_size)
        if not chunk:
            break
        buf = tail + chunk
        for m in pattern.finditer(buf):
            global_start = offset - len(tail) + m.start()
            print('match at', global_start)
        tail = buf[-overlap:]
        offset += len(chunk)

If you can memory-map the file, let the OS present the complete file as a single byte sequence and run re.finditer once; matches returned from that scan are already absolute offsets. See Python’s mmap docs for details: https://docs.python.org/3/library/mmap.html

A few extra notes: in Python 3 use byte patterns (b'...') when scanning binary data; if the maximum match length is unknown, either pick a safe overlap or use mmap/the third-party regex module (it supports incremental/partial matching) to avoid false negatives; on 32-bit processes mapping extremely large files can be limited by address space.

Recommended Answers

All 2 Replies

Member Avatar for Gribouillis

Why don't you write

for each in wp_head:
        print "Word Perfect header %s found @ offset %d" % (
                             each.group(), total_data_read + each.start())

?

Another issue to consider is that a match could occur between 2 blocks, for example \xFF\x57 at the end of a 4096 bytes block and \x50\x43 at the beginning of the next block. Currently, your code won't find the match.

Edited by Gribouillis because: n/a
Member Avatar for Banjoplucker

Thank you!

As usual, I had started looking at this long after my bedtime and had noted the total_data_read angle as my next line of attack to provide an accumalitive offset.I shall retire now otherwise I may well get my second wind and be up until the early hours!!

Kind Regards

Banjoplucker

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.