Ad:

Java Discussion Thread
Unsolved
Views: 21915

Aug 11th, 2005

Reading Large Files

Expand Post »

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

Java Syntax (Toggle Plain Text)

java.lang.OutofMemory

This is how i am trying to read the file

Java Syntax (Toggle Plain Text)
FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 
 
int len = 0;
byte[] buffer1 = new byte[1024]; 
 
while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();
FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 

int len = 0;
byte[] buffer1 = new byte[1024]; 

while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

Java Syntax (Toggle Plain Text)
System.gc();
System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

Similar Threads

Reading Binary Files in VB in Visual Basic 4 / 5 / 6
Trying to send large files overseas securely in Networking
downloading very large files with PHP in PHP
problems with reading video files in Windows NT / 2000 / XP
PHP for large files in PHP

freesoft_2000

Reputation Points: 25

Solved Threads: 10

Practically a Master Poster

Offline

623 posts
since Jun 2004

Permalink

Aug 11th, 2005

Re: Reading Large Files

Try flushing the buffer after every write.
The only thing I can think of that's eating up memory here is the outputbuffer. It may not be flushing automatically, causing the JVM to attempt to map the entire output into RAM instead of writing it directly to disk (which you seem to be attempting to do).

jwenting

Team Colleague

Reputation Points: 1658

Solved Threads: 331

duckman

Offline

7,719 posts
since Nov 2004

Permalink

Aug 12th, 2005

Re: Reading Large Files

Maybe you should try setting the maximum heap size for the JVM.

Java Syntax (Toggle Plain Text)

java -Xmx256m MyClass

You can put higher values if needed.
For more help, www.NeedProgrammingHelp.com

NPH

Reputation Points: 10

Solved Threads: 1

Junior Poster in Training

Offline

55 posts
since May 2005

Permalink

Aug 12th, 2005

Re: Reading Large Files

doesn't matter, having a 256MB heap won't help you contain a 600MB file in memory

You don't NEED to retain all that data in memory so the real solution is to not retain the data at all.

jwenting

Team Colleague

Reputation Points: 1658

Solved Threads: 331

duckman

Offline

7,719 posts
since Nov 2004

Permalink

Aug 12th, 2005

Re: Reading Large Files

Hi everyone,

Is this what you meant NPH

Java Syntax (Toggle Plain Text)

java -Xmx650m MyProgram

Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

I think maybe jwenting maybe right that i have to flush and i will be trying it out.

On another question do any of you guys know how to increase the heap size programatically?

Richard West

freesoft_2000

Reputation Points: 25

Solved Threads: 10

Practically a Master Poster

Offline

623 posts
since Jun 2004

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Java Forum Timeline: do while loop - not stopping!!

Next Thread in Java Forum Timeline: Can Anyone Help Me Please?

DaniWeb Message

Cancel Changes

User Name
Password