DaniWeb Forum Index > Software Development > C++

C++ RSS

Splitting a string into tokens

Please support our C++ advertiser: Intel Parallel Studio Home

•

Join Date: Dec 2008

Posts: 57

Reputation: AdRock is an unknown quantity at this point

Solved Threads: 0

AdRock

Offline

Junior Poster in Training

Splitting a string into tokens

33 Days Ago

I have a string and I need to split the string into tokens

The word can contain and letter/number but can also contain punctation such as brackets

I need to be able to find an occurence of punctuation, copy the string up to that point as a token, copy the puctuation mark as a token etc until it reaches the end of the string

Here is my output for the string dressed(with)

•

vector 0: dressed
vector 1: (with)
vector 2: dressed(with
vector 3: )

vectors 0 and 3 are correct but the output should be

•

vector 0: dressed
vector 1: (
vector 2: with
vector 3: )

here is my code

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <string>
#include <iostream>
#include <vector>
 
using namespace std;
 
int main()
{
	vector <string> vec;
	string str = "dressed(with)";
	string tmp;
 
	char punct[] = {'+','-','*','/','<','=','!','>','{','(',')','}',';',','};
 
	for (int i=0; i < sizeof(punct); i++)
	{
		unsigned int pos = str.find(punct[i], 0);
 
		if(pos != string::npos)
		{
			tmp.assign(str, 0, pos);
			vec.push_back(tmp);
			tmp.assign(str, pos, pos);
			vec.push_back(tmp);
		}	
	}
 
	for(int a=0; a < vec.size(); a++)
	{
		cout << "vector " << a << ": " << vec.at(a) << endl; 
	}
 
	return 0;
}
#include <string>
#include <iostream>
#include <vector>

using namespace std;

int main()
{
	vector <string> vec;
	string str = "dressed(with)";
	string tmp;
	
	char punct[] = {'+','-','*','/','<','=','!','>','{','(',')','}',';',','};

	for (int i=0; i < sizeof(punct); i++)
	{
		unsigned int pos = str.find(punct[i], 0);

		if(pos != string::npos)
		{
			tmp.assign(str, 0, pos);
			vec.push_back(tmp);
			tmp.assign(str, pos, pos);
			vec.push_back(tmp);
		}	
	}
	
	for(int a=0; a < vec.size(); a++)
	{
		cout << "vector " << a << ": " << vec.at(a) << endl; 
	}

	return 0;
}

•

Join Date: Jun 2009

Posts: 681

Reputation: Tom Gunn has much to be proud of

Solved Threads: 132

Tom Gunn

Offline

Practically a Master Poster

33 Days Ago

Your search always starts from position 0, that is why tokens are being duplicated. You need to skip over the extracted tokens as you go. Something like this:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <iostream>
#include <string>
#include <vector>
 
int main()
{
    using namespace std;
 
    string const punct = "+-*/<=!>{()};,";
 
    string str = "dressed(with)";
    string::size_type pos = 0;
    vector<string> vec;
 
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);
 
        if (end == pos) end = str.find_first_not_of(punct, pos);
 
        vec.push_back(str.substr(pos, end - pos));
        pos = end;
    }
 
    for (int a = 0; a < vec.size(); ++a)
    {
        cout << "vector " << a << ": " << vec.at(a) << '\n'; 
    }
 
    return 0;
}
#include <iostream>
#include <string>
#include <vector>

int main()
{
    using namespace std;

    string const punct = "+-*/<=!>{()};,";

    string str = "dressed(with)";
    string::size_type pos = 0;
    vector<string> vec;

    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);

        if (end == pos) end = str.find_first_not_of(punct, pos);

        vec.push_back(str.substr(pos, end - pos));
        pos = end;
    }

    for (int a = 0; a < vec.size(); ++a)
    {
        cout << "vector " << a << ": " << vec.at(a) << '\n'; 
    }

    return 0;
}

-Tommy (For Great Justice!) Gunn

•

Join Date: Jan 2008

Posts: 3,814

Reputation: VernonDozier has a reputation beyond repute

Solved Threads: 501

VernonDozier VernonDozier is offline

Offline

Senior Poster

33 Days Ago

If I recall correctly (from a previous thread), the OP also wants each punctuation element to be in its own vector (OP, please clarify), so if the string was "dressed(with))", the OP wants this:

dressed
(
with
)
)

rather than

dressed
(
with
))

so the OP would have to take it another step or two and further split strings with multiple consecutive punctuation characters into separate strings (again, that's if I was interpreting correctly from earlier threads). But Tom Gunn's code gets you one step closer regardless!

•

Join Date: Dec 2008

Posts: 57

Reputation: AdRock is an unknown quantity at this point

Solved Threads: 0

AdRock

Offline

Junior Poster in Training

33 Days Ago

•

Originally Posted by VernonDozier

If I recall correctly (from a previous thread), the OP also wants each punctuation element to be in its own vector (OP, please clarify), so if the string was "dressed(with))", the OP wants this:

dressed

(

with

)

)

rather than

dressed

(

with

))

so the OP would have to take it another step or two and further split strings with multiple consecutive punctuation characters into separate strings (again, that's if I was interpreting correctly from earlier threads). But Tom Gunn's code gets you one step closer regardless!

Yes....that is a good point are you are right. there may be an occurence where that would be needed. Every punctuation mark should be in it's own vector.

How would i rewrite that?

Last edited by AdRock; 33 Days Ago at 11:14 am.

•

Join Date: Jun 2009

Posts: 681

Reputation: Tom Gunn has much to be proud of

Solved Threads: 132

Tom Gunn

Offline

Practically a Master Poster

33 Days Ago

•

How would i rewrite that?

Give it a try before asking for help. Depending on your experience solving problems, at least an hour to several hours of solid work at it should be the minimum. Honestly, if somebody tells you how to write the code every time, you will not learn anything substantial and you will end up asking for help with everything.

-Tommy (For Great Justice!) Gunn

•

Join Date: Dec 2008

Posts: 57

Reputation: AdRock is an unknown quantity at this point

Solved Threads: 0

AdRock

Offline

Junior Poster in Training

33 Days Ago

Thanks

I've just solved another problem I've had for ages and that's splitting the strings of each line into tokens

This is where my current problem leads onto where i need to split each signle string into tokens

•

Join Date: Dec 2008

Posts: 57

Reputation: AdRock is an unknown quantity at this point

Solved Threads: 0

AdRock

Offline

Junior Poster in Training

32 Days Ago

I am struggling to come up with a solution for this as everything i have tried either gets the same output or the program crashes.

This is how i understand it

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
string::size_type end = str.find_first_of(punct, pos);
string::size_type end = str.find_first_of(punct, pos);

assign to the variable end where the first occurrence of any of the puncs starting from the first char

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
if (end == pos)
if (end == pos)

if the end variable is 0 then

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
end = str.find_first_not_of(punct, pos);
end = str.find_first_not_of(punct, pos);

assign to the variable end where the first occurrence of any non puncs starting from the first char

It then loops around starting at the new pos which using this string

•

(dressed(with))

which would be 0,18,9,13 but stops at 13

Do i have to perform another loop inside of the while loop and have the vec.push_back inside?

The way i thought it would be done is it finds a punc at pos whatever and then it should go through the while loop again

•

Join Date: Jan 2008

Posts: 3,814

Reputation: VernonDozier has a reputation beyond repute

Solved Threads: 501

VernonDozier VernonDozier is offline

Offline

Senior Poster

32 Days Ago

Here's what I think you should do. Don't worry for now about why/how the code did what it did (obviously, you can and probably should go over it later and figure out what it does and why for your own personal knowlege). Test it out with all sorts of input and make sure it does what it's supposed to (break strings into "all non-punctuation" and "all punctuation" strings). If it does work for all possible test cases, go to the next step.

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);
 
        if (end == pos) end = str.find_first_not_of(punct, pos);
 
        vec.push_back(str.substr(pos, end - pos));
        pos = end;
    }
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);

        if (end == pos) end = str.find_first_not_of(punct, pos);

        vec.push_back(str.substr(pos, end - pos));
        pos = end;
    }

Break line 7 into two lines:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
string newString = str.substr (pos, end - pos);
vec.push_back (newString);
string newString = str.substr (pos, end - pos);
vec.push_back (newString);

So you end up with this:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);
 
        if (end == pos) end = str.find_first_not_of(punct, pos);
 
        string newString = str.substr (pos, end - pos);
        vec.push_back (newString);
        pos = end;
    }
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);

        if (end == pos) end = str.find_first_not_of(punct, pos);

        string newString = str.substr (pos, end - pos);
        vec.push_back (newString);
        pos = end;
    }

Now, if newString contains punctuation, you need to change it into one string for every character. If it doesn't, push the whole string as you do in line 8 above. Test whether it has any punctuation in it, as before, and act accordingly (push it onto the vector if it's all non-punctuation, split it further if it is punctuation):

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
if (newString.find_first_of (punct) == string::npos)
{
   // newString doesn't contain punctuation. Push it.
   vec.push_back (newString);
}
else
{
   // newString is punctuation.  Break newString into one-character strings and push each of them onto vec.
}
if (newString.find_first_of (punct) == string::npos)
{
   // newString doesn't contain punctuation. Push it.
   vec.push_back (newString);
}
else
{
   // newString is punctuation.  Break newString into one-character strings and push each of them onto vec.
}

So your job is:

Try Tom Gunn's code out. Make sure it "works" for all possible test cases (i.e. change line 11 below for every possible test case you can think of and make sure the code "behaves". I imagine it does. Tom Gunn's code generally does. . But you need to verify that.
If it does, look at my revised code below. Run it. See what it does. now delete my line 31. Change line 30 so it does what you need it to do, which is to take a string like "****))" strored in newString and break it into six separate strings, and push them all onto the vec vector

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <iostream>
#include <string>
#include <vector>
 
int main()
{
    using namespace std;
 
    string const punct = "+-*/<=!>{()};,";
 
    string str = "dressed(to**!impress{{)";
    string::size_type pos = 0;
    vector<string> vec;
 
    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);
 
        if (end == pos) end = str.find_first_not_of(punct, pos);
 
        string newString = str.substr (pos, end - pos);
 
        if (newString.find_first_of (punct) == string::npos)
        {
            // newString doesn't contain punctuation. Push it.
            vec.push_back (newString);
        }
        else
        {
            // newString is punctuation.  Break newString into one-character strings and push each of them onto vec.
            vec.push_back ("PUNCTUATION");
        }
 
        pos = end;
    }
 
    for (int a = 0; a < vec.size(); ++a)
    {
        cout << "vector " << a << ": " << vec.at(a) << '\n';
    }
 
    return 0;
}
#include <iostream>
#include <string>
#include <vector>

int main()
{
    using namespace std;

    string const punct = "+-*/<=!>{()};,";

    string str = "dressed(to**!impress{{)";
    string::size_type pos = 0;
    vector<string> vec;

    while (pos != string::npos)
    {
        string::size_type end = str.find_first_of(punct, pos);

        if (end == pos) end = str.find_first_not_of(punct, pos);

        string newString = str.substr (pos, end - pos);

        if (newString.find_first_of (punct) == string::npos)
        {
            // newString doesn't contain punctuation. Push it.
            vec.push_back (newString);
        }
        else
        {
            // newString is punctuation.  Break newString into one-character strings and push each of them onto vec.
            vec.push_back ("PUNCTUATION");
        }

        pos = end;
    }

    for (int a = 0; a < vec.size(); ++a)
    {
        cout << "vector " << a << ": " << vec.at(a) << '\n';
    }

    return 0;
}

•

Join Date: Jun 2009

Posts: 681

Reputation: Tom Gunn has much to be proud of

Solved Threads: 132

Tom Gunn

Offline

Practically a Master Poster

32 Days Ago

•

Try Tom Gunn's code out. Make sure it "works" for all possible test cases (i.e. change line 11 below for every possible test case you can think of and make sure the code "behaves".

It does not, as you proved. I did not consider adjacent punctuation in my haste to get my post out the door and ended up over engineering the whole thing. Since punctuation is always a single character in this case, the simpler solution for matching punctuation works better:

 Help with Code Tags 
 C++ Syntax (Toggle Plain Text) 
#include <iostream>
#include <string>
#include <vector>
 
std::vector<std::string> SplitOnPunct(std::string const& str,
                                      std::string const& punct)
{
    std::vector<std::string> vec;
 
    if (str.length() == 0) return vec;
 
    std::string::size_type pos, end;
 
    for (pos = 0; pos != std::string::npos; pos = end)
    {
        end = str.find_first_of(punct, pos);
 
        if (end == pos && ++end == str.size()) end = std::string::npos;
 
        vec.push_back(str.substr(pos, end - pos));
    }
 
    return vec;
}
 
int main()
{
    std::vector<std::string> vec = SplitOnPunct("dressed(with)", "+-*/<=!>{()};,");
 
    for (std::vector<std::string>::size_type a = 0; a < vec.size(); ++a)
    {
        std::cout << "vector " << a << ": " << vec.at(a) << '\n'; 
    }
}
#include <iostream>
#include <string>
#include <vector>

std::vector<std::string> SplitOnPunct(std::string const& str,
                                      std::string const& punct)
{
    std::vector<std::string> vec;

    if (str.length() == 0) return vec;

    std::string::size_type pos, end;

    for (pos = 0; pos != std::string::npos; pos = end)
    {
        end = str.find_first_of(punct, pos);

        if (end == pos && ++end == str.size()) end = std::string::npos;

        vec.push_back(str.substr(pos, end - pos));
    }

    return vec;
}

int main()
{
    std::vector<std::string> vec = SplitOnPunct("dressed(with)", "+-*/<=!>{()};,");

    for (std::vector<std::string>::size_type a = 0; a < vec.size(); ++a)
    {
        std::cout << "vector " << a << ": " << vec.at(a) << '\n'; 
    }
}