How to split a large list

[squidd]

ok so basically I am writing a small program that does a few different things in Delphi 2007. I am very new to coding and so I hope that my terminology will be up to par. If not, please forgive me.

I am loading text files into a ListView... I know arrays can do this much faster, but I am trying to do this in a more visual way. When I load a list of oh, say 15,000 lines... I would like to be able to split the list and then save them into smaller lists. I have an editbox to display how many lines to use for splitting..

For example: I want to split the 15,000 line text file into 250 lines.. or maybe 3,000 lines. I cant for the life of me figure out how to do this.

Once the files are split, then naturally they need to be saved as well.. and that is the other problem.

ANY help on this would be a huge relief!

My code is as follows and is horribly wrong i am WELL aware:

procedure TForm1.Button12Click(Sender: TObject);
var
txt : textfile;
x,y : integer;
begin
savedialog1.Execute();
AssignFile(txt, saveDialog1.FileName + '.txt');
ReWrite(txt);
for x := 0 to ListView1.Items.Count - 1 do
begin
Writeln(txt, ListView1.Items[x].Caption, edit6.text);
end;
end;

As you can see, this is extremely fouled up and I have been trying for hours to figure this out. Thanks for any help in advance. :)

[Duoas]

The choice of TListView is a bit odd to me. You could have just as easily used a TListBox or TRichEdit component, which will give you an items: TStrings or lines: TStrings property, as well as LoadFromFile and SaveToFile methods.

You also need to check to make sure the user did actually choose a filename. The filename returned will always be the full path of the filename --there is no need to add '.txt' to it:

procedure TForm1.Button12Click(Sender: TObject);
  begin
  if savedialog1.Execute() then
    ListBox.items.saveToFile(saveDialog1.fileName)
  end;

If you only want to save, say, 250 lines at a time, you can do something like:

procedure save250( filename: string; var strs: tStrings; startAt: integer );
  var
    sl:   tStringList;
    b, e: integer;
  begin
  // Our temporary string list
  sl := tStringList.create;
  // indices of lines to copy, inclusive
  b  := startAt;
  e  := startAt + 249;
  if e >= strs.count then e := strs.count -1;
  // copy the indexed lines into our temporary string list
  for b := b to e do
    sl.append( strs[ b ] );
  // save the lines to file
  sl.saveToFile( filename );
  // cleanup
  sl.free
  end;

Did you really want the caption saved at the beginning of each line?

Hope this helps.

[squidd]

thank you very much for your quick reply.... :)

I am wanting to be able to say in an editbox the number of lines to save.. not just one number or the other. When I click the button that says to perform the procedure, I want the action to be:

1. count from the beginning of the original large list (aka text file) to whatever number is typed in the editbox.

2. save the files that are split into whatever name is chosen by the user named one after the other incrementally. ex: mytextfile1.txt, mytextfile2.txt, etc... until the entire list is split and saved in the desired directory. (mytextfile would be what the user specified in another editbox to give the file a base name)

so, enter how many lines you want to save to each text file in the edit6.text box. use that number to then save that amount of lines to each text file until the list that WAS one huge list, will now be smaller lists incrementally adding until is is complete.

Again, thank you very much for your help :) I hope that I have made this a bit clearer. if not, I apologize.

[Duoas]

Ah, that helps. You need to be more careful about reading the documentation when using various procedures (line writeln).

For what you want to do, the save250 procedure can almost do it. Just add another parameter for the number of lines to save and adjust line 10 to use the correct value.

Then you can save each n lines in a loop.

procedure TForm1.Button12Click( Sender: TObject );
  var
    lines_per_block: integer;
    line_index:      integer;
    file_number:     integer;
    file_extension:  string;
    base_filename:   string;
  begin
  lines_per_block := strToInt( edit6.text );  // number of lines to save per file
  if saveDialog1.execute then
    begin
    // get the file extension and everything except the extension in
    // separate strings so we can inject the 1, 2, 3, ... file number.
    file_extension := extractFileExt( saveDialog1.filename );
    base_filename := copy(
      saveDialog1.filename,
      1,
      length( saveDialog1.filename ) -length( file_extension )
      );
    // save each block of lines
    line_index  := 0;
    file_number := 1;
    while line_index < ListBox.items.count do
      begin
      save250(
        base_filename + intToStr( file_number ) + file_extension,
        ListBox.items,
        line_index,
        lines_per_block
        );
      inc( file_number );
      inc( line_index, lines_per_block )
      end
    end
  end;

I used the name save250 just because that is what I gave you before, but I imagine you will want to rename it. Also, you can make it a private member of the TForm if you like...

Hope this helps.

[squidd]

Thanks you very much.. it is late here and I will certainly pour over this to see if I can make it work for my purposes.

I will indeed reply and let you know either way.

Thank you for your prompt and friendly advice in my problem.

[squidd]

For whatever reason I cant get that procedure to properly compile to test it.. It keeps stopping on the save250 part with a complaint of undeclared indentifier.

I have tried a few things but none will compile for me either.

[Duoas]

Did you define a procedure named save250? It won't work unless you do...

[squidd]

as far as I know I have defined the procedure at the top of the code where the rest of the procedures are written. but it still wont compile for some reason. I am sure this is user error and not with what you have helped me with :(

thanks again for your fast replies! :)

[Duoas]

No compiler error is user error... not unless the 'user' is the person using the compiler. :$

Check that you spelled everything correctly, that everything is in the proper scope, and if you still can't get it to work post your code here and I'll take a look at it.

[squidd]

I added the code exactly as you put it in there...

I made some changes from listbox to listview, and defined the procedure up top.

procedure save250(sender: TObject);

procedure TForm1.Button12Click( Sender: TObject );
var
lines_per_block: integer;
line_index: integer;
file_number: integer;
file_extension: string;
base_filename: string;
begin

lines_per_block := strToInt( edit6.text ); // number of lines to save per file
if saveDialog1.execute then
begin
// get the file extension and everything except the extension in
// separate strings so we can inject the 1, 2, 3, ... file number.
file_extension := extractFileExt( saveDialog1.filename );
base_filename := copy( saveDialog1.filename,1,length( saveDialog1.filename ) -length( file_extension ));
// save each block of lines
line_index := 0;
file_number := 1;
while line_index < ListView1.items.count do
begin
save250(base_filename + intToStr( file_number ) + file_extension,
ListView1.items, line_index, lines_per_block);
inc( file_number );
inc( line_index, lines_per_block );
end;
end;
end;

/And here are the errors I am receiving when I try to compile.

[DCC Error] : E2065 Unsatisfied forward or external declaration: 'TForm1.save250'
[DCC Error] : E2034 Too many actual parameters
[DCC Error] : E2010 Incompatible types: 'TObject' and 'string'

thanks again :)

[Duoas]

You didn't define save250 the way I gave it to you. The one you defined takes a single argument: a TObject. The one I gave you takes entirely different arguments.

The procedure header should look like this: procedure SaveNLines( filename: string; var strs: tStrings; startAt, numLines: integer ); Please notice how I renamed the procedure from save250 to saveNLines.

Hope this helps.

[squidd]

sigh... i cant get this to compile... *bangs head against the wall*

I copy your exact line changed it to SaveNLines where needed, and back to save250, then to AGGHH... no avail...

[squidd]

Here is what I have now....

Procedure Header:
procedure SaveNLines( filename: string; var strs: tStrings; startAt, numLines: integer );

Code For Button12:
procedure TForm1.Button12Click(Sender: TObject);
var
lines_per_block: integer;
line_index: integer;
file_number: integer;
file_extension: string;
base_filename: string;
begin
lines_per_block := StrToInt( edit6.text ); // number of lines to save per file
if saveDialog1.execute then
begin
// get the file extension and everything except the extension in
// separate strings so we can inject the 1, 2, 3, ... file number.
file_extension := extractFileExt( saveDialog1.filename );
base_filename := copy( saveDialog1.filename,1,length( saveDialog1.filename ) -length( file_extension ));
// save each block of lines
line_index := 0;
file_number := 1;
while line_index < ListView1.items.count do
begin
SaveNLines(base_filename + intToStr( file_number ) + file_extension,
ListView1.items, line_index, lines_per_block);
inc( file_number );
inc( line_index, lines_per_block );
end;
end;
end;

And the errors that inevitably ensue when I attempt to compile:
[DCC Error] : E2065 Unsatisfied forward or external declaration: 'TForm1.SaveNLines'
[DCC Error] : E2033 Types of actual and formal var parameters must be identical

I simply have no idea why this is happening... I really appreciate your patience and help on this matter regardless of whether or not I can get this work for me. :)

[Duoas]

Don't bang your head too hard... you don't want to break it.

Unsatisfied forward or external declaration
You can declare a thing and you can define a thing:

unit fooey;
interface
// Everything in this section is a DECLARATION
function hum( bark: string ): string;

implementation
// Everything this section is a DEFINITION
function hum( bark: string ): string;
  begin
  result := 'Baz says "' + bark + '"'
  end;

end.

An Object Pascal unit requires you to first declare your functions, etc. in the interface section. Then you must use the implementation section to define the thing things you declared. If you do not define it, but only declare it, then you are basically telling the compiler that some function exists when it does not.

The type of things matters
Recall that I previously recommended you to use something other than a TListView? (such as a TListBox)? That is because the items property is a TListItems, not a TStrings. The former is weird and hard to handle. The latter is convenient and easy to handle. Unless you are making a spreadsheet or something you should not be using a TListView.

The procedure I gave you takes a TStrings, not a TListItems. If you insist on using a TListView then you will have to convert your items property into a TStrings object before using the function I gave you. If you don't want to do that either then you will have to get every string out of the items property and write it to file in a loop.

Tell me what you want to do and I'll help you proceed. But I'll also expect you to spend a little more time thinking about it first... Don't feel bad. I am not trying to make this hard for you. Honest.

Hope this helps.

[squidd]

I want to do this is in listview... Not listbox.. List box will be my next endeavor.

I thought I was clear on wanting this done in ListView at the very start. If not, my apologies. The whole program uses ListView and would require and enormous amount of time to switch everything to a ListBox.

I am new at this, and this one thing and remove duplicate entries are the only things left to complete this project. In listView, not Listbox.

Thank you for your help and patience with a newbie.

[Duoas]

Alright. Here's a SaveNLines that takes a TListItems.

procedure saveNLines(
  filename:  string;
  var items: tListItems;
  startAt:   integer;
  numLines:  integer
  );
  var
    sl:   tStringList;
    b, e: integer;
  begin
  // Our temporary string list
  sl := tStringList.create;
  // indices of lines to copy, inclusive
  b  := startAt;
  e  := startAt + numLines -1;
  if e >= items.count then e := items.count -1;
  // copy the indexed lines into our temporary string list
  for b := b to e do
    sl.append( items.item[ b ].caption );
  // save the lines to file
  sl.saveToFile( filename );
  // cleanup
  sl.free
  end;

Notice that nothing other than each item's caption is saved to file.

Sorry for the confusion. P.S. I haven't tested this code. It might need a tweak here or there.

[squidd]

And this will split files and begin a save sequence when I click button12?

It looks like it is a procedure that does not have the button being used at all. Let me give it a go however and thank you very much. :)

ok here is what I did and I think maybe a tweak or two may be needed as you mentioned earlier... I am glad this is so easy for you.. it sure isnt for me. :(

Added to types:

procedure saveNLines( filename: string; var items: tListItems; startAt: integer; numLines: integer );

Added to the button12 to perform the action of splitting files:
procedure TForm1.Button12Click(Sender: TObject);
var
sl: tStringList;
b, e: integer;
begin
// Our temporary string list
sl := tStringList.create;
// indices of lines to copy, inclusive
b := startAt;
e := startAt + numLines -1;
if e >= items.count then e := items.count -1;
// copy the indexed lines into our temporary string list
for b := b to e do
sl.append( items.item[ b ].caption );
// save the lines to file
sl.saveToFile( filename );
// cleanup
sl.free
end;
end;

And the resulting compile errors:

[DCC Error] : E2029 ')' expected but identifier 'item' found
[DCC Error] : E2003 Undeclared identifier: 'items'
[DCC Error] : E2003 Undeclared identifier: 'numLines'
[DCC Error] : E2003 Undeclared identifier: 'startAt'

Looks like this is getting much closer to working now even though there are errors... Also, what about my edit6.text being used to determine the number of lines to split the list into? Or am I missing something in what you coded?

[Duoas]

I'd like to help more but at this point you aren't paying attention. I've given you all the code you need --which is much more than I usually do. You have to use your brain now and use the code I've given you correctly.

[squidd]

uhm.. ok

thanks for your help

[squidd]

Anyone else to take a stab at solving this?

Seems this gentleman has given up...

In my defense, the code posted on page one was not correct at all and was to be used in a ListBox procedure (which probably isnt not a bad idea at all).. Regardless, once that was straightened out, I get more code that doesn't work. If I knew how to do this, I wouldn't be asking. While I appreciate the person's help and his apparent 'above and beyond' normal helping. The problem is no closer to being solved that when I first posted my question for help.

Thanks for any future help regarding this 'evidently' easy task. :)

[Duoas]

There's no need for the angry attitude or to get all defensive. I have far more experience than you and, whether you believe it or not, a greater understanding of your problem than you do.

At no point have I given you incorrect code --all the code I supplied works fine. The problem lies in your understanding.

May I ask how it is that you have written so much code that it would take an enormous amount of time to switch to a TListBox (which, you will note, I have not required you to do), without understanding the basic concepts central to programming? At no point have I written anything in any post of mine which does not directly apply to your efforts. Don't just skip over stuff you don't understand because you think it has no bearing.

You are perfectly capable of working with what I have given you. It is neither over your head nor is it smarter than you. You must, if you intend to continue working with Delphi, learn how to lookup information about the VCL with the Delphi help system, and you must understand how to create and use procedures and functions, and you must understand how the type of a thing affects the code used to manipulate it. Failing these things you will remain stuck.

It is time now to put a little effort into learning something about these things instead of demanding that I or anyone else simply give you code to cut and paste (which, BTW, I have already done). Programming is not easy. It takes a lot of work, and years to become sufficiently proficient that tasks like this become 'evident'.

I have already said, and remain, willing to help. But I am unwilling to do your work for you.

[squidd]

Wasnt getting defensive nor angry... You are not willing to help me get this one thing complete or understand what I am not doing correctly. I have spent hours on looking up information to get this done before posting this question here.

I am VERY new to programming... I am doing this alone on my own time at home. I never said you were not more experienced than me. So please. Just stop posting and turning this post into a completely unnecessary flame war.

Thanks (again) for your help. But it wasnt enough to help me understand why i cant get it to work. You are right, I am wrong. Now please let someone else with better communication skills try to help me out with this should they choose to. If I understood why I cant get it to work, then I wouldnt be typing still.

Have a nice day kind sir.

[Duoas]

Look, I haven't meant to make you feel stupid.

1. Look at posts #4 and #10, where I gave you an event handler and which you correctly modified to use the TListView.
2. Now look at post #16 where I gave you a function that prints x lines from a TListView's item list.
3. Put them both in your code. Change the word "save250" from post #10 to "saveNLines".

I know you are new to programming. That is why I have asked you to read about procedures and about types. All your errors are due as a direct consequence of your failure to do that.

You are being defensive and angry. And you presume too much to say I am unwilling or that I misunderstand. I understand better than you think, and I have stated repeatedly I am willing to help, if you are willing to follow the advice given you. I have no interest in wasting my time to lead you in circles.

Please consider that I know what it is to learn how to program and how much information overload it is to find simple answers. The truth is that there are no simple answers when it comes to programming, and you are going to have to wrap your mind around a few things --none of which come easily-- to get it right. Until you do, even a gift horse looks like a mule.

[squidd]

1. Look at posts #4 and #10, where I gave you an event handler and which you correctly modified to use the TListView.
2. Now look at post #16 where I gave you a function that prints x lines from a TListView's item list.
3. Put them both in your code. Change the word "save250" from post #10 to "saveNLines".

And again, after all this is done.. it wont compile.. still.

[DCC Error] : E2065 Unsatisfied forward or external declaration: 'TForm1.saveNLines'
[DCC Error] : E2197 Constant object cannot be passed as var parameter

Even after I did EXACTLY what you recommend, it doesnt compile.. has never compiled.

I simply don't know how to correct these errors. I cant make plainer than that.

EDIT:

Also, I have to say. If this is what I have to go through to learn one single thing for each thing I try to do, then I dont know how much longer I am going to bother with learning how to program. I have spent an ENORMOUS amount of time trying to figure this single procedure out, and the effort is simply not worth the payoff in this case i am starting to think. You cannot even begin to comprehend my frustration with this. I am getting REALLY pissed. Not at you so much, but at the idea that a simple thing such as this is so much trouble. Almost a week on this ONE thing.

[Duoas]

Yeah, I know. I've gone through the same things... Over and over and over again. It's just part of programming...

Post your entire unit and I'll take a look-see. It has to be something painfully silly.

I once spent an entire week on about 60-70 lines of C code and was just at the point of burning my PC when I figured out that I had misplaced a } somewhere. Most. Obnoxious. Error. Ever.

Things get better after a while. Honest they do. Just don't give up. (Right now I'm banging my own head against the windows console... the stupid thing doesn't have a WndProc, permit event hooks, anything... it is for all intents and purposes an actual GUI-less process. All I want to know is if the user grabs the window and sizes it... but apparently that "can't happen". Alas...)

I think you put the saveNLines inside your TForm1 class definition, but defined it below as procedure saveNLines( ... ); instead of procedure TForm1.saveNLines( ... ); I'm not sure what the second error is, so post everything so I can find it.

[squidd]

I don't think I need to post the entire program.. its almost 300 lines. Which I know is nothing, but it works fine without the code in question. See if you can find out what is wrong here first. If not Ill post the whole thing.

procedure Button12Click(Sender: TObject);  
procedure saveNLines(filename:  string; var items: tListItems; startAt: integer; numLines: integer);

private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject); 
begin
  Form1.Caption := 'List Modify';
  ListView1.Columns.Add.Caption := '0';
  Listview1.Columns[0].Width := 300;
end;

procedure TForm1.Button12Click( Sender: TObject );
var
lines_per_block: integer;
line_index: integer;
file_number: integer;
file_extension: string;
base_filename: string;
begin
lines_per_block := strToInt( edit6.text ); // number of lines to save per file
if saveDialog1.execute then
begin
// get the file extension and everything except the extension in
// separate strings so we can inject the 1, 2, 3, ... file number.
file_extension := extractFileExt( saveDialog1.filename );
base_filename := copy( saveDialog1.filename,1,length( saveDialog1.filename ) -length( file_extension ));
// save each block of lines
line_index := 0;
file_number := 1;
while line_index < ListView1.items.count do
begin
SaveNLines(base_filename + intToStr( file_number ) + file_extension,
ListView1.items, line_index, lines_per_block);
inc( file_number );
inc( line_index, lines_per_block );
end;
end;
end;

procedure TForm1.saveNLines(filename: string; var items: tListItems; startAt: integer; numLines: integer);
var
sl : tStringList;
b, e : integer;
begin
  // Our temporary string list
sl := tStringList.create;
  // indices of lines to copy, inclusive
b  := startAt;
e  := startAt + numLines -1;
if e >= items.count then e := items.count -1;
  // copy the indexed lines into our temporary string list
for b := b to e do
sl.append( items.item[ b ].caption );
  // save the lines to file
sl.saveToFile( filename );
  // cleanup
sl.free
end;

I hope you can find the problem.... I am bummed out. Thanks again :)

[Duoas]

OK, that was enough. You need two changes.

In your class definition:

private
    { Private declarations }
    procedure saveNLines(filename: string; items: tListItems; startAt: integer; numLines: integer);
  public
    { Public declarations }
  end;

You must declare your procedures and functions before you can use them. Also, the compiler was complaining about passing the TListView.items as a var argument, so you will notice that I removed the var from the parameter list. The actual procedure definition must match, so change it below the same way:

procedure TForm1.saveNLines(filename: string; items: tListItems; startAt: integer; numLines: integer);

That compiled and worked correctly for me.

[squidd]

yes SIR! It did compile! Thank you very much :)

But nothing happens when I add a number to the edit6.text box and click button12 to perform the now correct (thank you very much for that btw) code.

What is missing I wonder to where the button (button12) isnt doing the procedures being called upon?

I am happy at this point to just have it compile. But the splitting of a large list still doesnt happen .

I loaded up a list of over 15,000 lines. and clicked the button. Nothing happened. I say ok fine.. no problem. so I load up a list again and type in '20' in edit6.text. Then I went to the save button, and it didn't split it up there either.

maybe I should post the whole thing for you to peruse. This is insane. lol

[Duoas]

It works, it just doesn't tell you it did. Check your directory for files.

Remember also that the TListView must be populated with at least one item for it to work. (You loaded your list so that's not a problem.) When I did it, I just added three items using the designer, set the number of items per file to 2, and clicked the button --producing two files (the first containing two lines and the last containing one). I'm sure you must have a ton of files in the directory you saved to.

Most programs indicate that a save worked successfully by putting a message on the statusbar or changing the main form's caption or some other like thing.

Another thing you should be aware of is that there is currently no error checking to make sure that Edit6 contains a number, so if the user puts in "fooey" the program will crash. Go ahead and try it. There are several ways to fix this, but the easiest is something like:

try
  lines_per_block := strToInt( edit6.text )
except
  showMessage( 'You must specify how many lines to save per file.' );
  edit6.setFocus;
  exit
  end;
if saveDialog1.execute then ...

Glad to be of help.

[squidd]

it doesn't split the file nor does it save to the folder where the original file was loaded from. Actually, after added a value i.e. 50 to edit6.text, I then clicked button12 (after compiling all this first of course) and nothing happens.. Nothing is being split. If it is, then why isnt it saving?

Is that another thing altogether? Thanks again.

and yes, error checking is definately something I will need to do and possibly will help with, but no need for that until i get this to work. lol