1,105,271 Community Members

Reading Large Files

Member Avatar
freesoft_2000
Practically a Master Poster
623 posts since Jun 2004
Reputation Points: 9 [?]
Q&As Helped to Solve: 11 [?]
Skill Endorsements: 0 [?]
 
0
 

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

java.lang.OutofMemory

This is how i am trying to read the file

FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 

int len = 0;
byte[] buffer1 = new byte[1024]; 

while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

Member Avatar
jwenting
duckman
8,333 posts since Nov 2004
Reputation Points: 1,399 [?]
Q&As Helped to Solve: 445 [?]
Skill Endorsements: 35 [?]
Infraction Points: 5
Team Colleague
 
0
 

Try flushing the buffer after every write.
The only thing I can think of that's eating up memory here is the outputbuffer. It may not be flushing automatically, causing the JVM to attempt to map the entire output into RAM instead of writing it directly to disk (which you seem to be attempting to do).

Member Avatar
NPH
Junior Poster in Training
55 posts since May 2005
Reputation Points: 0 [?]
Q&As Helped to Solve: 1 [?]
Skill Endorsements: 0 [?]
 
0
 

Maybe you should try setting the maximum heap size for the JVM.

java -Xmx256m MyClass

You can put higher values if needed.
For more help, www.NeedProgrammingHelp.com

Member Avatar
jwenting
duckman
8,333 posts since Nov 2004
Reputation Points: 1,399 [?]
Q&As Helped to Solve: 445 [?]
Skill Endorsements: 35 [?]
Infraction Points: 5
Team Colleague
 
0
 

doesn't matter, having a 256MB heap won't help you contain a 600MB file in memory :)
You don't NEED to retain all that data in memory so the real solution is to not retain the data at all.

Member Avatar
freesoft_2000
Practically a Master Poster
623 posts since Jun 2004
Reputation Points: 9 [?]
Q&As Helped to Solve: 11 [?]
Skill Endorsements: 0 [?]
 
0
 

Hi everyone,

Is this what you meant NPH

java -Xmx650m MyProgram

Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

I think maybe jwenting maybe right that i have to flush and i will be trying it out.

On another question do any of you guys know how to increase the heap size programatically?

Richard West

You
This article has been dead for over three months: Start a new discussion instead
Post:
Start New Discussion
Tags Related to this Article