>>is it possible to use mmap for this.
Sure it's possible, but is it the best solution? mmap is a general one-size-fits-all solution for large data structures that are paged. It is most likely that your application can take advantage of the structure of your matrix and its usage to have a better scheme for swapping between RAM and disk.
You need to essentially look at your algorithms in terms of parallelism and see what parts of the matrix are used at a given time. For example, say your operation is a simple matrix multiplication A*B, where both A and B are very large. Then a "good" scheme might be to store A as row-major in a file and store B as column-major in another file. To do the multiplication you would do:
//say A is in fileA, B is in fileB, and C = A*B is in fileC.
for_each_row_i_of_A
load_next_row(A,fileA);
for_each_column_j_of_B {
load_next_column(B,fileB);
C[j] = 0.0;
for(int k=0;k<N;++k)
C[j] += A[k]*B[k];
};
save_row(C,fileC);
};
The above is appropriate because you have the minimum amount of loading and saving operations and all loading and saving are continuous blocks of memory (because A and C are row-major and B is column-major). You could have the same algorithm with A and C as column-major and B as row-major but the loading and saving operations would be much more expensive because you would have to load element by element instead of the entire row or column at once.
Of …