Is this a code parser? I see you are looking for "/*" and "*/", which are comment delimiters. if not, what is the significance of the tests on lines 29 through 33? Since whitespace is a delimiter and not just newline, I'd go with the >> operator rather than getline. >> separates all whitespace for you.
I'm going to the Brown's house for dinner.
Read in using >> and it splits to this:
I'm
going
to
the
Bob's
house
for
dinner.
Next task with each word is to split into punctuation. I'd go through character by character looking for punctuation and grabbing their indexes.
Character at index 3 is punctuation, indexes are 0 through 4, so three substrings:
Bob // up to but not including index 3.
' // index 3
s // everything after index 3
Stick the three strings in the array, go on to the next word. Decide exactly what "punctuation" is and go through a character at a time. If punctuation is what is defined in "ispunct", use it. Otherwise write your own. If the "tokens" are more complicated than single characters, this approach won't work.