Java

Java RSS

Reading Large Files

•

Join Date: Jun 2004

Posts: 609

Reputation: freesoft_2000 is an unknown quantity at this point

Solved Threads: 7

freesoft_2000 freesoft_2000 is offline

Offline

Practically a Master Poster

Reading Large Files

Aug 11th, 2005

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

Help with Code Tags

Java Syntax (Toggle Plain Text)

java.lang.OutofMemory

This is how i am trying to read the file

 Help with Code Tags 
 Java Syntax (Toggle Plain Text) 
FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 
 
int len = 0;
byte[] buffer1 = new byte[1024]; 
 
while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();
FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 

int len = 0;
byte[] buffer1 = new byte[1024]; 

while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

 Help with Code Tags 
 Java Syntax (Toggle Plain Text) 
System.gc();
System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

Microsoft uses "One World, One Web, One Program" as a slogan.
Doesn’t that sound like "Ein Volk, Ein Reich, Ein Führer" to you, too?
— Eric S. Raymond

Tell me what type of software do you like and what would you pay for it

http://www.daniweb.com/techtalkforums/thread19660.html

•

Join Date: Nov 2004

Posts: 6,143

Reputation: jwenting is just really nice

Solved Threads: 213

jwenting

Offline

duckman

Re: Reading Large Files

Aug 11th, 2005

Try flushing the buffer after every write.
The only thing I can think of that's eating up memory here is the outputbuffer. It may not be flushing automatically, causing the JVM to attempt to map the entire output into RAM instead of writing it directly to disk (which you seem to be attempting to do).

As people are clearly allowed to attack me but I'm not allowed to defend myself, I no longer post to this site.

•

Join Date: May 2005

Posts: 55

Reputation: NPH is an unknown quantity at this point

Solved Threads: 1

NPH

Offline

Junior Poster in Training

Re: Reading Large Files

Aug 12th, 2005

Maybe you should try setting the maximum heap size for the JVM.

Help with Code Tags

Java Syntax (Toggle Plain Text)

java -Xmx256m MyClass

You can put higher values if needed.
For more help, www.NeedProgrammingHelp.com

•

Join Date: Nov 2004

Posts: 6,143

Reputation: jwenting is just really nice

Solved Threads: 213

jwenting

Offline

duckman

Re: Reading Large Files

Aug 12th, 2005

doesn't matter, having a 256MB heap won't help you contain a 600MB file in memory

You don't NEED to retain all that data in memory so the real solution is to not retain the data at all.

As people are clearly allowed to attack me but I'm not allowed to defend myself, I no longer post to this site.

•

Join Date: Jun 2004

Posts: 609

Reputation: freesoft_2000 is an unknown quantity at this point

Solved Threads: 7

freesoft_2000 freesoft_2000 is offline

Offline

Practically a Master Poster

Re: Reading Large Files

Aug 12th, 2005

Hi everyone,

Is this what you meant NPH

Help with Code Tags

Java Syntax (Toggle Plain Text)

java -Xmx650m MyProgram

Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

I think maybe jwenting maybe right that i have to flush and i will be trying it out.

On another question do any of you guys know how to increase the heap size programatically?

Richard West