Data Structure > Main Memory

[kng]

So guys for the sake of simplicity lets say i am developing a huge matrix multiplication program using C and i made a data structure which is able to represent this huge matrix into miniature blocks the down side is that this representation of block takes much more memory than i got RAM.

The question
What would be the most efficient way to deal with this problem.
- Currently I am trying to further compress my data structure but i doubt i will be able to fit it into main memory.
- I am currently thinking of writing blocks back to hard disk but this doesn't sound that efficient any idea's ?

[jephthah]

you probably want to look into sparse matrix compression algorithms

[kng]

Just for the book i am applying a ton of compression techniques but lets just say for some reason we still can't be able to fit the entire structure onto main memory what would you do?

[Ancient Dragon]

Just for the book i am applying a ton of compression techniques but lets just say for some reason we still can't be able to fit the entire structure onto main memory what would you do?

you might have at least a couple choices

1. Use a 64-bit compiler which gives the program more memory. But that's won't help if the program already maxed out the computer's RAM.
2. Swap the data to disk. This is a bad solution because it will be very very sloooooow.
3. Replace the array with an SQL database table.
4. Re-design the program so that it's not such a memory hog.

[abhimanipal]

When we multiply 2 matrices, each element of the first row of the first matrix is multiplied with each element of the first column of the second matrix and so on
If you can bring one row of the first matrix and one column of the second matrix into the main memory, multiply them and store the result back to the disk, this might speed up the computation