improve performance of the following io codes

Reply

Join Date: Oct 2005
Posts: 22
Reputation: kimw is an unknown quantity at this point 
Solved Threads: 0
kimw kimw is offline Offline
Newbie Poster

improve performance of the following io codes

 
0
  #1
Oct 18th, 2006
can anyone offer performance tips to improve the running time ?
this function opens a file (a 7000 row by 30 col), and stores each elements in a matrix data . current running time is 4 sec, and i desperately need to minimize the running time as i need to iterate thru
thousands of such files please help, thanks

  1. void iocsv(vector<vector<double> >& data, string path)
  2. {
  3. string s;
  4. ifstream inFile;
  5.  
  6. inFile.open(path.c_str());
  7. if (inFile) {
  8. while (getline(inFile, s)) {
  9. vector<double> col;
  10. tokenizer<escaped_list_separator<char> > tok(s);
  11. for (tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end(); ++beg) {
  12. istringstream price;
  13. price.str(*beg);
  14. double x;
  15. price >> x;
  16. col.push_back(x);
  17. }
  18. data.push_back(col);
  19. }
  20. } else { cerr << "Warning: cannot open file " << path << endl;
  21. cerr << "Program terminating ......" << endl; }
  22.  
  23. inFile.close();
  24. }
Reply With Quote Quick reply to this message  
Join Date: Aug 2005
Posts: 5,264
Reputation: iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold iamthwee is a splendid one to behold 
Solved Threads: 377
Featured Poster
iamthwee's Avatar
iamthwee iamthwee is offline Offline
Posting Expert

Re: improve performance of the following io codes

 
0
  #2
Oct 18th, 2006
Originally Posted by kimw View Post
can anyone offer performance tips to improve the running time ?
this function opens a file (a 7000 row by 30 col), and stores each elements in a matrix data . current running time is 4 sec, and i desperately need to minimize the running time as i need to iterate thru
thousands of such files please help, thanks

  1. void iocsv(vector<vector<double> >& data, string path)
  2. {
  3. string s;
  4. ifstream inFile;
  5.  
  6. inFile.open(path.c_str());
  7. if (inFile) {
  8. while (getline(inFile, s)) {
  9. vector<double> col;
  10. tokenizer<escaped_list_separator<char> > tok(s);
  11. for (tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end(); ++beg) {
  12. istringstream price;
  13. price.str(*beg);
  14. double x;
  15. price >> x;
  16. col.push_back(x);
  17. }
  18. data.push_back(col);
  19. }
  20. } else { cerr << "Warning: cannot open file " << path << endl;
  21. cerr << "Program terminating ......" << endl; }
  22.  
  23. inFile.close();
  24. }
Are the files sorted? The reason why I ask is because although the inital sorting costs a lot in terms of time, once sorted it would be a lot quicker to retrieve data.
*Voted best profile in the world*
Reply With Quote Quick reply to this message  
Join Date: Oct 2005
Posts: 22
Reputation: kimw is an unknown quantity at this point 
Solved Threads: 0
kimw kimw is offline Offline
Newbie Poster

Re: improve performance of the following io codes

 
0
  #3
Oct 18th, 2006
hi the file is not sorted ... although i could build a macro to sort all the files (they are all in csv format) but i wish to see what other alternatives there are
Reply With Quote Quick reply to this message  
Join Date: Dec 2005
Posts: 5,850
Reputation: Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute Salem has a reputation beyond repute 
Solved Threads: 749
Team Colleague
Salem's Avatar
Salem Salem is offline Offline
Void main'ers are DOOMed

Re: improve performance of the following io codes

 
0
  #4
Oct 18th, 2006
How long does an empty loop take?
  1. while (getline(inFile, s)) {
  2. }
Separate the "time to read the file" from the time to "tokenise the file".

It it takes <1 second, then there might be something you can do.

If it takes >3 seconds, then all your tokenising/vector stuff is not the problem.

> and i desperately need to minimize the running time as i need to iterate thru
> thousands of such files
Or just not worry about it and let the program run overnight, and it will all be done by morning anyway. If that can be done, it certainly isn't worth spending more than a day trying to make it vastly more efficient.
By your measure, it's about 900 files per hour.
Last edited by Salem; Oct 18th, 2006 at 4:45 pm.
Reply With Quote Quick reply to this message  
Join Date: Oct 2005
Posts: 22
Reputation: kimw is an unknown quantity at this point 
Solved Threads: 0
kimw kimw is offline Offline
Newbie Poster

Re: improve performance of the following io codes

 
0
  #5
Oct 18th, 2006
it was good suggestion, well i've tested the emply loop and it took < 1 sec so it must be all the tokenizer and vector stuffs ... am i going overboard by using tokenizer, since i basically wanted to store all the elements in csv file into a matrix
Reply With Quote Quick reply to this message  
Reply

This thread is more than three months old.
Perhaps start a new thread instead?
Message:



Similar Threads
Other Threads in the C Forum
Thread Tools Search this Thread



About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC