Reading Large Files - Memory Management

[phillipdaw]

I am attempting to read and process a very large .csv (~75 Mb). I have been able to load the file into a string, but when I begin processing the string I run into problems. I am looking for memory management advice on these sorts of operations.

String.Split() - OutOfMemoryException, which did not suprise me.
Manual split using 'for loops' slows the comp to a crawl.

My coworker wrote a perl script to accomplish the task and monitored his memory usage, which approached 1 gig (which is all my work computer has).

I have tried replacing strings with stringbuilders to reduce overhead, but I eventually have to go to strings to use the split function and run into the same problems.

[kvprajapati]

you may try to use System.IO.StringRead object and read this object line by line and split it. I think this whould be faster ... I am guessing.

[phillipdaw]

That might help... Currently I am attempting to convert the data to float as I split, to reduce the number of immutable objects in memory. I was converting the file into a string array, then splitting the data between string and float arrays later in the program.