malloc fails after 7000 to 8000 loops - available memory is 1GB

[gilly_kumar]

Hi,

After 7000-8000 loops, i.e after 2-3 days, there is a runtime error because malloc failed.We identified that the error was occuring malloc function tires to allocate ~250k bytes of data.

Then on analysis, We listed few possible items where malloc could fail

1. Fragmentation
2. Memory overwrite or overflow
3. Memory Leak
4. memory size was more than the memory available

malloc takes the value of 262144 bytes to allocate.So it is a valid value which is less than 1GB. Available memory is ~1GB.
So, based on the data we ruled out 3rd and 4th item.

But Fragmentation, we are not sure like, because how Fragmentation creates the problem even though available memory is 1GB. Also, every run we release a memory as it is evident that there 1GB.

Memory overwrite or overflow, if it had happened, it should have crashed initially itself. But we get this error only after ~7000-10000 device run.

I could not guess anything beyond this, and i dont know how to proceed on this. This is an huge application runs at customer site, where we dont have an access to try and see or gather the data.

It is something, i need to go theorically and proove with the sample program and then try to fix it.

Please help me in this Guyz...

thanks in advance.
Kumar

[Ancient Dragon]

Years ago we used Microsoft VC++ 1.52C compiler. We had a similar problem and found that malloc() itself cause memory leaks every few hundred calls. We wrote our own malloc() function and the problem disappeared.

You might write a small test program to see if malloc() is doing something similar with your compiler and operating system.

[gilly_kumar]

Years ago we used Microsoft VC++ 1.52C compiler. We had a similar problem and found that malloc() itself cause memory leaks every few hundred calls. We wrote our own malloc() function and the problem disappeared.

You might write a small test program to see if malloc() is doing something similar with your compiler and operating system.

It is actually an 32 bit C/C++ compiler and the version is 12.0.8804.0. I tried checking the memory leak with the sample program through the Performance monitor, but did not find any leak.

- Senthil

[Salem]

What is your basis for ruling out a memory leak?

250K * 8000 is 2GB, which is close to the upper limit of what common 32-bit operating systems allocate to the user address space.

> Memory overwrite or overflow, if it had happened, it should have crashed initially itself.
Whilst this is likely, the absence of a crash is not evidence for absence of a bug.

Using tools like valgrind and electric fence (or purify if you're on windows) tells you more.

Which OS/Compiler (and versions) are you running with?

[Ancient Dragon]

>>It is actually an 32 bit C/C++ compiler
What compiler are you using and on what operating system ?

Are there other system resources that are being consumed and not released, such as file handles, or graphics gui objects.

[gilly_kumar]

What is your basis for ruling out a memory leak?

250K * 8000 is 2GB, which is close to the upper limit of what common 32-bit operating systems allocate to the user address space.

> Memory overwrite or overflow, if it had happened, it should have crashed initially itself.
Whilst this is likely, the absence of a crash is not evidence for absence of a bug.

Using tools like valgrind and electric fence (or purify if you're on windows) tells you more.

Which OS/Compiler (and versions) are you running with?

>>Which OS/Compiler (and versions) are you running with?
Windows XP

>What is your basis for ruling out a memory leak?
Since we could see that the available memory is of size 1GB, at that particular moment, when malloc fails.

thanks for the Reply
- Senthil

[gilly_kumar]

>>It is actually an 32 bit C/C++ compiler
What compiler are you using and on what operating system ?

Are there other system resources that are being consumed and not released, such as file handles, or graphics gui objects.

No, that is the only application being run in production environment.

thanks for viewing it..
- Senthil

[gilly_kumar]

What is your basis for ruling out a memory leak?

250K * 8000 is 2GB, which is close to the upper limit of what common 32-bit operating systems allocate to the user address space.

> Memory overwrite or overflow, if it had happened, it should have crashed initially itself.
Whilst this is likely, the absence of a crash is not evidence for absence of a bug.

Using tools like valgrind and electric fence (or purify if you're on windows) tells you more.

Which OS/Compiler (and versions) are you running with?

>>250K * 8000 is 2GB, which is close to the upper limit of what common 32-bit operating systems allocate to the user address space.

So, you mean, 250k is always left as memory leak, i.e every run.. is that correct. But actually it is been released, since there is 1GB available memory.

Thanks
Senthil

[jephthah]

i would never use malloc() where longterm runtime stability is critical.

that said, i don't understand how you can so easily rule out a memory leak.

yes, i hear what you're saying about how your available memory appears to be ~1GB... but to use that sort of indirect observation to necessarily rule out a leak, well... that's a dangerous assumption.

as Salem so insightfully noted, 8000 * 256K = 2G ... now THAT is highly suspicious. One might even go as far to say that is the sort of evidence for a memory leak.

at any rate, to so cavalierly rule out a memory leak off the top is IMHO crippling your ability to discover root cause.

.

[Salem]

Process explorer can tell you about different kinds of memory allocation.

For example, it's possible to have say 1GB allocated to the process, but only 100MB actually mapped in memory at that moment. PE can show you both of these figures.

I would suggest you instrument your malloc/free calls so you have a trace of allocation (and amount) and free calls. You should then be able to simulate in a few seconds what it takes days to do normally.

Also get to use perfmon, which can be used to record performance statistics (eg. mapped and allocated memory) over an extended period of time. Perhaps the memory use is level and jumps unexpectedly in response to certain events.

[gilly_kumar]

Process explorer can tell you about different kinds of memory allocation.

For example, it's possible to have say 1GB allocated to the process, but only 100MB actually mapped in memory at that moment. PE can show you both of these figures.

I would suggest you instrument your malloc/free calls so you have a trace of allocation (and amount) and free calls. You should then be able to simulate in a few seconds what it takes days to do normally.

Also get to use perfmon, which can be used to record performance statistics (eg. mapped and allocated memory) over an extended period of time. Perhaps the memory use is level and jumps unexpectedly in response to certain events.

Thanks for the Reply.

I have got some info from the log we have like,

Well, I have got the information like,

Available System Memory: 1438244 Kb during the start of the program and at the end, when the error was reported

Available System Memory: 957008 Kb

-------------------

481236 KB ~ 500MB

Yes, i agree, there is a memory leak. But i have a doubt like, you have told that this process would have 2GB of max, if that is the case then at the leak of 500MB will that leads to malloc fail?

- Senthil

[jephthah]

i agree, there is a memory leak.

disappointing i know. you so wanted it to be either a flaw in MSVC++ or perhaps in malloc itself. I know, ive been there.

but take this as a lesson, right now, to quit using malloc in programs where the need for extended runtime is critical .... or better yet, quit using it at all.

it doenst make you a better programmer because you can use it. in fact, it tends to be inexperienced and/or sloppy programmers who use malloc most often. 99% of the time i see malloc used, it is completely unnecessary and could be just as easily avoided, and half the time it's implemented poorly.

this is not 1995 any more. RAM is cheap and readily available. if you just absolutely have to use malloc, use it only in tightly controlled situations.

.

[Salem]

> Available System Memory: 1438244 Kb during the start of the program and at the end, when the error was reported
> Available System Memory: 957008 Kb
You have a swap file, and lots of other processes on your machine. Depending on what everything else is up to, the OS will decide that the system risks becoming slow/unstable if resources become too low.

Also, 2GB is what your program could address (it's half the 4GB of a 32-bit address space). That doesn't mean you're going to be able to allocate that much, but does represent an upper limit.

Simulating your allocation pattern in a test program should tell you whether the run-time allocator is up to the job.

If you're always allocating and freeing approximately the same sized block, consider just allocating something comfortably bigger just once at program start up and then keep hold of it thereafter.

[Asat232]

More probable:
1. Memory Leak.
2. Memory improper usage, bad logic of our application.

To see it you can use performance statistics. Memory should constantly grow.
Applications used in production environment must be tested on different OS using for example virtual machines.
For debugging purposes you can run server on VM and client on your real
developer machine and vise versa.
Better trace your application then do thinking if malloc may leak.

[lich]

interesting thoughts... my group member used this Malloc Function, malloc and Alloc for a project...