954,505 Members — Technology Publication meets Social Media
Username:
Password:
Lost login information?
Have something to say? Contribute New Article Reply to this Article

Big files bring segmentation faults

/* One file can be hundreds of MBs. So, 640 MB in maximum*/
int nMaxByteCount = 64000000;

I read files to get binary info in it. But I don't know the exact sizes before reading. So, I allocate nMaxByteCount for large files first and get support from fread return to get the exact size.

*nFileSize = fread(caFileContent,sizeof(char),nMaxByteCount,DataFile);


Then I use this *nFileSize -which contain the exact file size- for filling information in my data structures. When I create a char array or struct containing these variables, I mostly get segmentation faults after reading big files more than 10 MBs. I think the allocated size of memory which the system gives access to my program is not suitable or enough for my files and further programming. How much size of bytes or KBs or MBs are permitted for my Linux C program on a 64 bit processor? Can I change it? Or, is this the problem?

asilter
Junior Poster in Training
60 posts since Oct 2006
Reputation Points: 10
Solved Threads: 0
 

why are you reading the file? All you have to do is call fseek() to move the file pointer to the end of the file then call ftell() to get the file size. Another way is call fstat() which returns the file size in a structure.

>>How much size of bytes or KBs or MBs are permitted for my Linux C program on a 64 bit processor
To my knowledge its limited only by memory.

You don't want to read that large files all into memory at one time anyway because the system will just swap the majority of it to disk and you have accomplished nothing. Its better to design your program so that you only need to read small chunks at a time, such as one record at a time.

Ancient Dragon
Retired & Loving It
Team Colleague
30,049 posts since Aug 2005
Reputation Points: 5,662
Solved Threads: 2,343
 

Given your previous posts, I'd say it's down to bugs in your code.

All things being equal, what you describe might be unwise for the reasons AD states, but it certainly shouldn't segfault.

Convince yourself by writing a simple program consisting only of your large array, and a simple main() containing only fopen() and fread().

Oh, and as for the rest of your code, I would suggest you do either of these
gcc -o prog prog.c
valgrind prog

or gcc -g -o prog prog.c -lefence
gdb prog
run

Both of these can tell you a lot about any mis-use of memory which doesn't immediately cause a segfault, but which is a problem non the less.

Salem
Posting Sage
Team Colleague
11,531 posts since Dec 2005
Reputation Points: 5,862
Solved Threads: 953
 

This article has been dead for over three months

Post: Markdown Syntax: Formatting Help
You