Member Avatar for techgig

how to read a file which contains more than 95000 <95 thousand> lines in java.

Edited by techgig because: n/a
  • 5 Contributors
  • 6 Replies
  • 205 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by jclawson000

Dani AI

Generated

Building on 's JVM tip and / 's streaming suggestion: the core question is whether the program actually needs every line in memory at once. 95K lines is not huge by itself; an OutOfMemoryError usually means either (a) each line is very large, (b) the code accumulates lines (List, single huge StringBuffer/StringBuilder), or (c) other objects retain references. The most robust fix is to stop keeping the whole file in RAM.

For straightforward, line-by-line processing use a streaming reader so each line is processed and discarded. Example (Java 8+):

Path p = Paths.get("largefile.txt");
try (Stream<String> lines = Files.lines(p, StandardCharsets.UTF_8)) {
    lines.forEach(line -> {
        // lightweight processing: parse, write result, aggregate counts, etc.
        // avoid adding 'line' to any long-lived collection
    });
}

If the program truly must keep the full dataset (sort, random access, or complex in-memory joins), use a disk-backed approach instead of a single large heap allocation. Two proven patterns:

  • External merge sort: split the file into memory-sized chunks, sort each chunk, write sorted chunks to temp files, then perform a k-way merge.
  • Use an embedded store (SQLite, H2, or a lightweight key-value file) to index or query records rather than storing all Strings in Java objects.

Troubleshooting tips: profile heap usage (VisualVM, jmap/jcmd) to see which objects dominate memory; avoid Scanner for heavy IO; prefer buffered readers or Files.lines; reuse buffers and avoid building very large StringBuilders. Increasing heap is a short-term workaround; the long-term, scalable solution is streaming or disk-backed processing so line count does not dictate memory use.

Recommended Answers

All 6 Replies

Member Avatar for ~s.o.s~

What kind of problems are you running into? If you can read a couple of lines, you can read 95K lines...

Member Avatar for techgig

When i doing this job i got this type of error

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)

Member Avatar for ~s.o.s~

As I suspected, you are running out of heap space. When spawning the JVM process, pass in the following switch as JVM argument: -Xms128m -Xmx128m .

If you are invoking the java process from the command line, you can use:

java -Xms128m -Xmx128m pkg.MainClass

If you are using an IDE like Netbeans or Eclipse, refer the relevant documentation (i.e. google for "set heap size netbeans/eclipse").

Another variant of this exception is when you run out of permgen (permanent generation) space. Read for more details.

Member Avatar for masijade

And think about whether you actually need all "95000 lines" in memeory at the same time. You would be better off reading a line, processing that line, reading the next line. Only rarely do you actually need all of the data in memory at the same time.

Member Avatar for tamaris

That's true. Line by line with a BufferedReader should be the way to go.

Member Avatar for jclawson000

I agree with masijade and tamaris. Maybe the OP can shed more light on the original problem and we can provide a more detailed algorithm to deal with his issue.

But anyways, S.O.S thanks for the info. It will probably be useful at some point in the future (to me, anyways).

Jake Clawson

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.