help parse csv file - a Matlab user's view

[by_stander]

Hey everyone,

I have a Matlab code that parses data from a csv file, performs some calculations on the data, and spits it back out into a new csv file. the new csv file contains all the data from the original file, with the new calculations appended on at the end. I need to write a C++ document that performs the same actions.

I'm new to working with C++, but do have limited experience working with it. I can do everything regarding the calculations, but getting the data into C++ then writing back to a new csv file is a point of frustration for me.

Any suggestions about how to go about doing this? Or links to helpful reading material?

TIA

[alexchen]

Post all your code. Maybe I can help.

[alexchen]

This might help.

[ravenous]

You can read in files using fstream objects. I posted something about this a couple of days ago here (you have to scroll down a bit to get to the part where the original poster has got their point across). Give that a go and see how you get on. This is probably the best way to go if you're starting out.

Personally, I like using Qt for this, since the QString class has a whole bunch of useful methods for manipulating text, particularly the QString::split() method, which enables you to do things like

QStringList points = fileStream.readLine().split(',', QString::SkipEmptyParts);

This reads a line from the file and splits it into the chunks that are separated by commas (in this case). But, if you're new to C++, maybe you shouldn't dice with Qt just yet :o)

[by_stander]

You can read in files using fstream objects. I posted something about this a couple of days ago here (you have to scroll down a bit to get to the part where the original poster has got their point across). Give that a go and see how you get on. This is probably the best way to go if you're starting out.

Personally, I like using Qt for this, since the QString class has a whole bunch of useful methods for manipulating text, particularly the QString::split() method, which enables you to do things like

QStringList points = fileStream.readLine().split(',', QString::SkipEmptyParts);

This reads a line from the file and splits it into the chunks that are separated by commas (in this case). But, if you're new to C++, maybe you shouldn't dice with Qt just yet :o)

Thanks! I had been playing around with fstream , and I guess my question now comes to is there any way to put the data from my csv file into an array or something like that? in matlab, I could parse the data field into a separate character array for the header, and a matrix for the data content. that made it pretty easy to call the data I needed and manipulate it that way.

I have many (~500) rows of data and can have anywhere between 15 and 80 columns of data

[ravenous]

There's no C++ "readMyData()" function, it's too low level for that. You can, however make your own function to do this quite easily. You need to open the file and read in a line (as in the link above), use the std::string::find() method to find the positions of the delimiters and then chop out the numbers by using std::string::substr() . This will get you strings that are numbers, so you need to make or use some function that converts them to numbers from strings. You can use atof() for this, if you want a rough and ready solution.

How you go about the details of storing the data is up to you, I would use a std::vector since I find the std::vector::push_back() method convenient, but some people feel that's over-kill and that you should use arrays. This is fine, but you'd have to take the extra step of establishing how big your data set is before you reserve the memory for it. This isn't a big deal, you just have to make two passes through the file; one to find the maximum number of columns and the number of rows in the file. You then assign the memory and then make a second pass through the file to read data into your newly created arrays. This sounds like a pain but it's not really, and your files seem like they're pretty small (I sometimes have files that are millions of lines of data). It's even less of a pain if all your rows have the same number of columns. In this case, you just have to find the number of columns from the first row and then just count the number of lines, which is a lot faster than going through each line and counting the delimiters.

Anyway, hope that helps a little.

[by_stander]

I think I got it started, here is the code I came up with for anyone who sees this thread.

So far all it does is looks at the header and stores each header into an element of a vector, and counts how many rows and how many columns are in the data file.

Please excuse any poor syntax... it still needs work

Thanks for the help ravenous! I think I can take it from here.

int main()
{
    string line, str;
    size_t pos_beg, pos_end;
    vector<string> header;

    ifstream f_id;

    f_id.open ("filename.csv", ios::in | ios::app);
    if (f_id.is_open())
    {
        {   // this code set provides the header
            getline (f_id, line);
            pos_beg = 0;
            pos_end = line.find(",");
            //cout << pos_beg << "," << pos_end << endl;

            str = line.substr(pos_beg, (pos_end-pos_beg));
            header.assign (1,str);
            //cout << header.back() << endl;

            for (pos_beg = pos_end; pos_end <= line.size() ; pos_beg = pos_end)
            {

                pos_beg = line.find(",", pos_end);
                pos_end = line.find(",", pos_beg+1);

                str = line.substr(pos_beg+1, (pos_end-pos_beg-1));
                header.push_back (str);

                //cout << pos_beg << "," << pos_end << endl;
                //cout << header.back() << endl;
            }

        }
        // the following code set provides the amount of rows
        int nrows = 0;
        while ( !f_id.eof() )
        {
            getline (f_id, line);
            nrows++;
        }

        // the following code set provides the number of columns

        int ncols;
        {
            for (int i = 0; header[i] != header.back(); i++)
            {
                ncols = i+1;
            }

        cout << ncols << endl << header.back() << endl << header[ncols] << endl;

        }

        f_id.close();
    }

[by_stander]

Well, I thought I had it covered.. I hit another road block. I can successfully cycle through each row of my data and put an individual row into a vector<string>.

I can't figure out how to convert from a vector<string> to a vector<double>. I am using atof() incorrectly.. just for the life of me can't figure out how.

Another problem I am having (likely won't be difficult after converting from vector<string> to vector<double>) is assigning the vector to an array, so I will have all of my data stored in one central matrix.

Help? Suggestions?

double data[nrows][ncols];
vector<string> row;
vector<double> row_d;

for (int i = 0; i < nrows-1; i++) // for loop cycles through rows successfully. need to figure out how to store them
        {
            getline (f_id, line);
            pos_beg = 0;
            pos_end = line.find(",");

            str = line.substr(pos_beg, (pos_end-pos_beg));
            row.push_back (str);

            for (pos_beg = pos_end; pos_end <= line.size() ; pos_beg = pos_end)
            {
                pos_beg = line.find(",", pos_end);
                pos_end = line.find(",", pos_beg+1);

                str = line.substr(pos_beg+1, (pos_end-pos_beg-1));
                row.push_back (str);
            }

            for (int k = 0; k < row.size(); k++)
            {
                row_d[k] = atof(row[k]); // HERE is the error, converting from string to double
            }

            for (int k = 0; k < row_d.size(); k++) // probably incorrect as well
            {
                data[i][k] = row_d[k];
            }

        }

[ravenous]

you need to make sure you have the <cstdlib> header included and also atof() accepts char * as an argument, not a std::string so you have to use the std::string::c_str() method to convert the std::string to a C-style char array:

atof(row[k].c_str());

[jonsca]

A stringstream (http://www.cplusplus.com/reference/iostream/stringstream/) would work also.