 |  |  |  |  |  |  |  |
Allocating Memory for a huge array
Please support our C++ advertiser: Intel Parallel Studio Home
 |
Hello,
I am having trouble with the memory size of an array I am using. At the moment I have an array to pointers of objects. These objects get read in from a file and sorted, then a new file is outputted after some filtering is done on the array.
The code I have set up works great up to about a 500 megabyte file. After that though I get a memory access error when I try to create the array.
That is how I am creating my array.
Is there any way to make a huge array or will I have to look into using hard disk space as a memory alternative?
Cameron
I am having trouble with the memory size of an array I am using. At the moment I have an array to pointers of objects. These objects get read in from a file and sorted, then a new file is outputted after some filtering is done on the array.
The code I have set up works great up to about a 500 megabyte file. After that though I get a memory access error when I try to create the array.
C++ Syntax (Toggle Plain Text)
That is how I am creating my array.
Is there any way to make a huge array or will I have to look into using hard disk space as a memory alternative?
Cameron
You may be trying to allocate too large a chunk of memory at once.
Have you considered using an STL std::deque, or a std::list (or other linked list)?
Is it absolutely necessary to load the entire list at once?
I presume you are doing some post-processing on the lidar data (that this isn't some sort of real-time system)?
Have you considered using an STL std::deque, or a std::list (or other linked list)?
Is it absolutely necessary to load the entire list at once?
I presume you are doing some post-processing on the lidar data (that this isn't some sort of real-time system)?
Last edited by Duoas; Jan 15th, 2008 at 11:12 pm.
•
•
Join Date: Dec 2005
Posts: 5,850
Reputation: 
Solved Threads: 751
> Lidar * DataSet = new Lidar[header->number_of_point_records];
What is the value of sizeof(Lidar) ?
What is the value of header->number_of_point_records
If you multiply them together, how close to 2GB do you get?
> or will I have to look into using hard disk space as a memory alternative?
Or think of ways of dealing with the file in chunks of a few hundred MB at a time.
This may involve writing out temporary files, but that depends on what you're doing.
What is the value of sizeof(Lidar) ?
What is the value of header->number_of_point_records
If you multiply them together, how close to 2GB do you get?
> or will I have to look into using hard disk space as a memory alternative?
Or think of ways of dealing with the file in chunks of a few hundred MB at a time.
This may involve writing out temporary files, but that depends on what you're doing.
•
•
•
•
Originally Posted by Duoas 
You may be trying to allocate too large a chunk of memory at once.
Have you considered using an STL std::deque, or a std::list (or other linked list)?
Is it absolutely necessary to load the entire list at once?
I presume you are doing some post-processing on the lidar data (that this isn't some sort of real-time system)?
•
•
•
•
Originally Posted by Salem
> Lidar * DataSet = new Lidar[header->number_of_point_records];
What is the value of sizeof(Lidar) ?
What is the value of header->number_of_point_records
If you multiply them together, how close to 2GB do you get?
> or will I have to look into using hard disk space as a memory alternative?
Or think of ways of dealing with the file in chunks of a few hundred MB at a time.
This may involve writing out temporary files, but that depends on what you're doing.
header->number_of_point_records depends on the file.
Beyond that I am sure it goes above 2 gig on some of my larger files, any advice on how to use temporary files and still be able to sort the whole array?
Cameron
•
•
Join Date: Dec 2005
Posts: 5,850
Reputation:
Solved Threads: 751
If all you're trying to do is sort the data, then look up merge sort.
Maybe I am being dense, but I look at the merge sort and I still see myself needing to pull in all the data into one large array eventually. In turn this seems like it would not work, but again, maybe I am being dense.
Cameron
Cameron
I think it would be helpful to know a little more about the Lidar data record. Presumably it is some large number and two coordinates (elevation, longitude, latitude)?
What do you mean by "sorted"? Sorted by elevation?
In any case, you need a divide-and-conquer algorithm for sorting.
What do you mean by "sorted"? Sorted by elevation?
In any case, you need a divide-and-conquer algorithm for sorting.
•
•
Join Date: Dec 2005
Posts: 5,850
Reputation:
Solved Threads: 751
Merge sorting creates N temporary files, each of which is sorted.
The actual merge then only needs to read ONE record at a time from each file to determine the final sort ordering.
The actual merge then only needs to read ONE record at a time from each file to determine the final sort ordering.
•
•
•
•
Originally Posted by Duoas
I think it would be helpful to know a little more about the Lidar data record. Presumably it is some large number and two coordinates (elevation, longitude, latitude)?
What do you mean by "sorted"? Sorted by elevation?
In any case, you need a divide-and-conquer algorithm for sorting.
The actual lidar data itself contains X, Y, Z, intensity, class, flightline, return, and timestamp data.
•
•
•
•
Originally Posted by Salem
Merge sorting creates N temporary files, each of which is sorted.
The actual merge then only needs to read ONE record at a time from each file to determine the final sort ordering.
Cameron
|
Other Threads in the C++ Forum
- Previous Thread: template class
- Next Thread: DLL problem
Views: 2525 | Replies: 8
| Thread Tools | Search this Thread |
Latest C++ Posts
- Today's Posts
- Unanswered Threads
Related Forum Features
Tag cloud for C++
6 add api array arrays beginner binary bitmap c++ c/c++ calculator char class classes code compile compiler console conversion convert count data delete desktop directshow dll encryption error file forms fstream function functions game getline givemetehcodez google graph homeworkhelper iamthwee ifstream input int integer java lazy lib linkedlist linux loop looping loops map math matrix memory microsoft newbie news node number output parameter pointer problem program programming project proxy python random read recursion recursive reference return sort string strings struct studio system template templates test text tree unix url variable vector video visual visualstudio win32 windows winsock word wordfrequency wxwidgets
