943,982 Members | Top Members by Rank

Ad:
  • C Discussion Thread
  • Unsolved
  • Views: 3003
  • C RSS
Oct 18th, 2006
0

improve performance of the following io codes

Expand Post »
can anyone offer performance tips to improve the running time ?
this function opens a file (a 7000 row by 30 col), and stores each elements in a matrix data . current running time is 4 sec, and i desperately need to minimize the running time as i need to iterate thru
thousands of such files please help, thanks

  1. void iocsv(vector<vector<double> >& data, string path)
  2. {
  3. string s;
  4. ifstream inFile;
  5.  
  6. inFile.open(path.c_str());
  7. if (inFile) {
  8. while (getline(inFile, s)) {
  9. vector<double> col;
  10. tokenizer<escaped_list_separator<char> > tok(s);
  11. for (tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end(); ++beg) {
  12. istringstream price;
  13. price.str(*beg);
  14. double x;
  15. price >> x;
  16. col.push_back(x);
  17. }
  18. data.push_back(col);
  19. }
  20. } else { cerr << "Warning: cannot open file " << path << endl;
  21. cerr << "Program terminating ......" << endl; }
  22.  
  23. inFile.close();
  24. }
Similar Threads
Reputation Points: 44
Solved Threads: 0
Newbie Poster
kimw is offline Offline
22 posts
since Oct 2005
Oct 18th, 2006
0

Re: improve performance of the following io codes

Click to Expand / Collapse  Quote originally posted by kimw ...
can anyone offer performance tips to improve the running time ?
this function opens a file (a 7000 row by 30 col), and stores each elements in a matrix data . current running time is 4 sec, and i desperately need to minimize the running time as i need to iterate thru
thousands of such files please help, thanks

  1. void iocsv(vector<vector<double> >& data, string path)
  2. {
  3. string s;
  4. ifstream inFile;
  5.  
  6. inFile.open(path.c_str());
  7. if (inFile) {
  8. while (getline(inFile, s)) {
  9. vector<double> col;
  10. tokenizer<escaped_list_separator<char> > tok(s);
  11. for (tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end(); ++beg) {
  12. istringstream price;
  13. price.str(*beg);
  14. double x;
  15. price >> x;
  16. col.push_back(x);
  17. }
  18. data.push_back(col);
  19. }
  20. } else { cerr << "Warning: cannot open file " << path << endl;
  21. cerr << "Program terminating ......" << endl; }
  22.  
  23. inFile.close();
  24. }
Are the files sorted? The reason why I ask is because although the inital sorting costs a lot in terms of time, once sorted it would be a lot quicker to retrieve data.
Featured Poster
Reputation Points: 1536
Solved Threads: 431
Posting Expert
iamthwee is offline Offline
5,865 posts
since Aug 2005
Oct 18th, 2006
0

Re: improve performance of the following io codes

hi the file is not sorted ... although i could build a macro to sort all the files (they are all in csv format) but i wish to see what other alternatives there are
Reputation Points: 44
Solved Threads: 0
Newbie Poster
kimw is offline Offline
22 posts
since Oct 2005
Oct 18th, 2006
0

Re: improve performance of the following io codes

How long does an empty loop take?
  1. while (getline(inFile, s)) {
  2. }
Separate the "time to read the file" from the time to "tokenise the file".

It it takes <1 second, then there might be something you can do.

If it takes >3 seconds, then all your tokenising/vector stuff is not the problem.

> and i desperately need to minimize the running time as i need to iterate thru
> thousands of such files
Or just not worry about it and let the program run overnight, and it will all be done by morning anyway. If that can be done, it certainly isn't worth spending more than a day trying to make it vastly more efficient.
By your measure, it's about 900 files per hour.
Last edited by Salem; Oct 18th, 2006 at 4:45 pm.
Team Colleague
Reputation Points: 5862
Solved Threads: 950
Posting Sage
Salem is offline Offline
7,164 posts
since Dec 2005
Oct 18th, 2006
0

Re: improve performance of the following io codes

it was good suggestion, well i've tested the emply loop and it took < 1 sec so it must be all the tokenizer and vector stuffs ... am i going overboard by using tokenizer, since i basically wanted to store all the elements in csv file into a matrix
Reputation Points: 44
Solved Threads: 0
Newbie Poster
kimw is offline Offline
22 posts
since Oct 2005

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in C Forum Timeline: draw diagnola ellipse
Next Thread in C Forum Timeline: representing a quadratic equation





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC