This is currently something in design phase, but I'd like to have the idea checked over.
The situation is roughly as follows:
I've got a display that can show some amount of structs with varying byte counts per struct.
I've got a big binary file, over a gigabyte in size, filled with data that is parsed into structs.
Make the entire file viewable with (seemingly) minimal process time.
My current solution is the following:
To both sides of the currently viewed structs (ie, previously viewed structs and the next structs to be viewed), buffer the structs in memory and only performing operations on the buffered structs.
As the view moves, there are defined 'buffer thresholds' where, if the view reaches the threshold, the program deallocates memory for some number of buffered structs that the view is 'moving away from' and then shifts the remaining structs into the 'cleaned space'. The buffer reads in structs from the file, and then waits for a threshold to be passed.
So far, so good. I'm thinking a vector would be a good choice for the buffer. Advancing through the file (scrolling down) is not a problem.
But scrolling up is.
Initially, I thought that it would be all right to rewind() and come back down from the start of the file.
However, as the files get longer, running from the start to the desired location is frankly insane and wastes a lot of time. There needs to be a way to move the buffer back through memory.
Fseek looks good, but it requires a long (number of bytes between the location and the origin) to go back. No problem?
Problem: The number of bytes in each struct is not constant. Going on bytecount alone without struct reference means that the whole process gets mubar (gotta think of the kiddies).
That makes me think about a dynamically-allocated set of longs that are set with byte values at the end of particular numbers of structs, allowing for the file to return to different positions based on SEEK_SET.
But that limits file size to 2 gigabytes, and I need to be able to scroll through more information than that. I need to be able to open files of, let's say, 20 gigabytes and be able to transverse the whole thing (if it were possible) as easily going up as going down, at least, from the user's perspective.
That's my idea. Is it any good as a starting point?
If not, anyone have suggestions on how to accomplish this? For example, pointers to positions in the file after an arbitrary number of structs that conveniently fit buffer sizes?
Thank you for your time.