external sort

[Diamonddrake]

I have hit a slump, my line sorting text program runs out of memory when sorting large files, the answer is using an external merge sort. basically it breaks the file into chunks, sorts them, then loads chunks of the chunks, and sorts them, then some how comes up with a fully sorted file.

I have searched the internet, with multiple search engines, for hours the past couple of days. I can't seem to find any help what so ever on this topic, I have found facts that it exists, its purpose reiterated hundreds of times, but no information on how to do it, not even a link to a book I could buy.

Im not a student trying to cheat, or a developer trying to make millions on a program. I write freeware apps, and this one is being created out of necessity, for myself and some of the company I keep.

Please Help!!!

[LizR]

Is the data you're sorting confidential in anyway? eg, if I asked you to PM me a link to a copy of your source and the data thats failing.. would you be prepared to place it somewhere I could get at it, so I could take a look?

[Diamonddrake]

the data I am using is a data file from a fiend of mines employer, it was the only suitable file for testing, and I am afraid it is confidential. but its just a large csv file. the code for the sorting method I posted here. all i can tell is that i need to use external sorting to handle large files. but i can't see to find any information on how to implement it.

[LizR]

OK, but I was more after the entire code, so I could recreate the issue.. A segment is not sufficient to do that.

If you could post a link to the code and how to create a similar but fake csv file (eg how many fields, how many lines, data types if necessary to make the sort work) I can take a look

[Diamonddrake]

pm me your email addy and I will send it to you as an attachment.

[stefansavev]

I've written a fast external sort:

http://www.codeproject.com/KB/recipes/fast_external_sort.aspx

Feel free to use it.