Trouble finding enough RAM for malloc() in Vista 64?

[Kadence]

I'm having a lot of trouble finding enough RAM for a program I'm writing on Vista 64 Ultimate system, even though the system has 8GB RAM and only 2GB is used.

I tried to see how many 10 million byte blocks I could allocate using malloc(), and the compiled program crashes after the 209th block. So it crashes after just around 1.94GB of total memory allocated, despite the system having multiple more gigs of RAM available.

I don't know if the issue has to do with contiguous RAM or what. The same test program doesn't segmentation fault on a Linux system until after the 320th block, which should be because it's hitting the 3GB kernel limit for a process.

I've tried using various compilers, including the command line compiler for Visual Studio 2008, which I assume is a 64 bit compiler.

The actual program I'm working on needs a ton of memory for memory mapping data files about 800MB in size, and malloc() large arrays of the same size for numerical calculations. I actually moved from Linux to Windows because it was 64 bit with 8GB of memory.

What can I do to be able to use more RAM with malloc()?

[Ancient Dragon]

>>Visual Studio 2008, which I assume is a 64 bit compiler.
No -- its a 32-bit compiler. But the Pro version can compile 64-bit code if you provide the right options. Just because you compile something on a 64-bit os doesn't mean you will automatically get a 64-bit program.

Post a short example program that produces the problem you are talking about.

[Ancient Dragon]

I just tested a short program and it stopped at about 2Gig just as you had also reported. I believe the reason is a 32-bit program can only access 2 Gig RAM.

To get the 64-bit version of VC++ 2008 you have to select an option during installation. Here is how to do it.

[Kadence]

(edit: whoops, you posted as I was reading that vcvarsall stuff :) I'll check out that link)

Thanks. The version I have is the Professional Edition. How do I use the 64-bit compiler? Do I change a setting somehow and use 'cl', or is there a flag to pass? I read How to: Enable a 64-Bit Visual C++ Toolset at the Command Line but didn't get how to actually use vcvarsall.bat.

Here's the code for my test program. It tries to allocate lots of arrays, and then assign a value to an element of the array (it's the assignment that causes the crash). The program crashes on my PC after the test207-test209, varying on different executions (not sure what that indicates). On Linux it seg faults after 320.

#include <iostream>

using namespace std;

int main(int argc, char *argv[]){
	double* test1 = (double*) malloc(10000000); test1[0] = 1.0; cout<<"Done 1"<<endl;
	double* test2 = (double*) malloc(10000000); test2[0] = 1.0; cout<<"Done 2"<<endl;
	double* test3 = (double*) malloc(10000000); test3[0] = 1.0; cout<<"Done 3"<<endl;
	...
	double* test398 = (double*) malloc(10000000); test398[0] = 1.0; cout<<"Done 398"<<endl;
	double* test399 = (double*) malloc(10000000); test399[0] = 1.0; cout<<"Done 399"<<endl;
	double* test400 = (double*) malloc(10000000); test400[0] = 1.0; cout<<"Done 400"<<endl;
}

[Ancient Dragon]

I don't know how to enable 64-bit. From what I see in the link I posted, and read in other places, its enabled during installation of the compiler. There is apparently an installation option that lets you install the 64-bit compiler. You may have to reinstall the compiler in order to get it.

[wildgoose]

All my Windows work is still under Win32.

Remember in 32-bit land you only have about 3.2Gig totally available to your application, which really works out to about 2Gig because Dev Studio is an oinker! It's a memory hog! Especially in a debug build.

You're using C++ compilers and using C memory allocation!

You should be using new/delete new[]/delete[]. Not Malloc Free.

For fun try GlobalAlloc(), under Vista 64. see how much it lets you allocate!

[Ancient Dragon]

>>which really works out to about 2Gig because Dev Studio

What is Dev Studio? You mean Dev-C++? In any event it has nothing to do with the problem and does not affect the amount of memory a program can use, unless the computer has less than 2 Gig of RAM and is out of hard drive swap space.

[wildgoose]

Sorry I meant Visual Studio. (Dev Studio is a nickname for it!)
And Windows swaps out memory as you say, but DevStudio is the memory hog. Windows gives memory to the more active application and Visual Studio is a memory hog when it runs. Totally Shut down Visual Studio and run your test again as a standalone application.

Better yet, Reboot, then run your program and check your numbers!

[Nick Evan]

Sorry I meant Visual Studio. (Dev Studio is a nickname for it!)

Never heard of that one?

Anyway: the real question is: Why would you ever want to use 2gb (!) of memory? I bet there's a better way to solve your problem then using 2gb of ram

[Kadence]

It was a 32 bit/64 bit issue. I had a ton of problems updating Visual Studio (got some real weird errors and have given up for now), but I was able to get my test program working using MinGW-64. Thanks for the help :)

Anyway: the real question is: Why would you ever want to use 2gb (!) of memory? I bet there's a better way to solve your problem then using 2gb of ram

It's a huge analytical data set. The data set alone is 1.2GB and needs to be memory mapped, the analytical matrices require more.

[jbennet]

Sorry I meant Visual Studio. (Dev Studio is a nickname for it!)

He has a point. Version 5 and 6 were called MS Developer Studio in many manuals and thats why the path for Visual Studio 5 and 6 was %programfiles%/devstudio