Hi,
I would like to ask if anyone know how can I create a tokenizer for a txt file in C++.
I find it difficult because there are not only words but there are also numbers and <p id> tags.

I have attached the file that is needed to be tokenized.

Could anyone help me ?

Thanks a lot

The file wasn't attached.

Maybe that's the point ...

you could use the boost tokenizer library. here are a few links:
http://www.boost.org/libs/tokenizer/index.html
http://www-eleves-isia.cma.fr/docume...r/examples.cpp

you could also use the boost string algorithms library (if the file is read line by line into a string)
http://www.boost.org/doc/html/string_algo.html

The file is something like that :

<P ID=1>
CONTENTS
</P>
<P ID=2>
VOLUME I
</P>
<P ID=3>
BOOK FIRST.--A JUST MAN
</P>
<P ID=4>
CHAPTER
I. M. Myriel
II. M. Myriel becomes M. Welcome
III. A Hard Bishopric for a Good Bishop
IV. Works corresponding to Words
V. Monseigneur Bienvenu made his Cassocks last too long
VI. Who guarded his House for him
VII. Cravatte
VIII. Philosophy after Drinking
IX. The Brother as depicted by the Sister
X. The Bishop in the Presence of an Unknown Light
XI. A Restriction
XII. The Solitude of Monseigneur Welcome
XIII. What he believed
XIV. What he thought
</P>
<P ID=5>
BOOK SECOND.--THE FALL
</P>
<P ID=6>
I. The Evening of a Day of Walking
II. Prudence counselled to Wisdom
III. The Heroism of Passive Obedience
IV. Details concerning the Cheese-Dairies of Pontarlier
V. Tranquillity
VI. Jean Valjean
VII. The Interior of Despair
VIII. Billows and Shadows
IX. New Troubles
X. The Man aroused
XI. What he does
XII. The Bishop works
XIII. Little Gervais
</P>