Memory Intensive - Memory or File

[phorce]

I'm working on a project that generates roughly between 13x512 values in which I compare at run-time. So the program outflow will be sort of as follows:

• Program executes
• Extracts (1) 12x512
• Extracts (2) 12x512
• Compares (1) to (2)
• Extract (3)
• Compares (3) to (2) (1)

The problem is, I don't know whether or not the quickest method would be to store the extracted values inside a text file and then read them in, or, just store inside memory and if I am storing inside memory, is there a quick way to access these? Such as memory mapping?

Thanks

[Moschops]

What's the size of each value? Are these simple int or double or the like, or some custom class?

If you're generating these values and then operating on them, writing them to a text file and then reading them back in again is massively expensive in time and pretty pointless. If you can hold them all in memory, do.

A memory map, as the term is commonly used in your scenario, is a way of quickly and efficiently working with a file of data, so if you're not using a file of data, it's not appropriate.

I'd suggest simply sticking the data in an array.

[phorce]

Hey, thank you for the reply. The values are just simple double values with nothing seriously data-intensive about them.

The program will be running for 20 minutes or so, depending on how long each recoding is for. So I can just dump the values that have not had any matches after a certain period. I just don't want to store massive amounts of data inside memory and for it to dramatically slow the program down?

Thanks

[Moschops]

Things will only slow down if you've got so much data that you overwhelm your available RAM and the OS starts storing it on the hard drive in virtual memory.

Ignoring any optimisations like keeping data in a register or in the CPU cache, it takes the same amount of time to read data from anywhere in memory. Simply using a lot of RAM doesn't affect execution speed.

I'm sure there are various crazy edge cases and lots of riders on that like how much memory your OS likes to use before it starts using the HDD for extra storage and all that sort of thing, but 512 double values will take up on the order of 4 kilobytes. Given that a typical desktop PC has memory measured in the GB, 4 kilobytes is trivial. Using a millionth of your RAM for this data won't slow anything down.