Tokenizer with <p id> tags

[katerinaaa]

Hi,
I would like to ask if anyone know how can I create a tokenizer for a txt file in C++.
I find it difficult because there are not only words but there are also numbers and tags.

I have attached the file that is needed to be tokenized.

Could anyone help me ?

Thanks a lot

[Narue]

The file wasn't attached.

[twomers]

Maybe that's the point ...

[vijayan121]

you could use the boost tokenizer library. here are a few links:
http://www.boost.org/libs/tokenizer/index.html
http://www-eleves-isia.cma.fr/documentation/BoostDoc/boost_1_29_0/libs/tokenizer/examples.cpp

you could also use the boost string algorithms library (if the file is read line by line into a string)
http://www.boost.org/doc/html/string_algo.html

[katerinaaa]

The file is something like that :

CONTENTS

VOLUME I

BOOK FIRST.--A JUST MAN

CHAPTER
I. M. Myriel
II. M. Myriel becomes M. Welcome
III. A Hard Bishopric for a Good Bishop
IV. Works corresponding to Words
V. Monseigneur Bienvenu made his Cassocks last too long
VI. Who guarded his House for him
VII. Cravatte
VIII. Philosophy after Drinking
IX. The Brother as depicted by the Sister
X. The Bishop in the Presence of an Unknown Light
XI. A Restriction
XII. The Solitude of Monseigneur Welcome
XIII. What he believed
XIV. What he thought

BOOK SECOND.--THE FALL

I. The Evening of a Day of Walking
II. Prudence counselled to Wisdom
III. The Heroism of Passive Obedience
IV. Details concerning the Cheese-Dairies of Pontarlier
V. Tranquillity
VI. Jean Valjean
VII. The Interior of Despair
VIII. Billows and Shadows
IX. New Troubles
X. The Man aroused
XI. What he does
XII. The Bishop works
XIII. Little Gervais