943,724 Members | Top Members by Rank

Ad:
  • C++ Discussion Thread
  • Unsolved
  • Views: 947
  • C++ RSS
May 31st, 2007
0

Tokenizer with <p id> tags

Expand Post »
Hi,
I would like to ask if anyone know how can I create a tokenizer for a txt file in C++.
I find it difficult because there are not only words but there are also numbers and <p id> tags.

I have attached the file that is needed to be tokenized.

Could anyone help me ?

Thanks a lot
Similar Threads
Reputation Points: 10
Solved Threads: 0
Newbie Poster
katerinaaa is offline Offline
11 posts
since May 2007
May 31st, 2007
0

Re: Tokenizer with <p id> tags

The file wasn't attached.
Administrator
Reputation Points: 6442
Solved Threads: 1393
Bad Cop
Narue is offline Offline
11,807 posts
since Sep 2004
May 31st, 2007
0

Re: Tokenizer with <p id> tags

Maybe that's the point ...
Reputation Points: 453
Solved Threads: 57
Posting Virtuoso
twomers is offline Offline
1,873 posts
since May 2007
Jun 1st, 2007
0

Re: Tokenizer with <p id> tags

you could use the boost tokenizer library. here are a few links:
http://www.boost.org/libs/tokenizer/index.html
http://www-eleves-isia.cma.fr/docume...r/examples.cpp

you could also use the boost string algorithms library (if the file is read line by line into a string)
http://www.boost.org/doc/html/string_algo.html
Reputation Points: 1159
Solved Threads: 285
Posting Virtuoso
vijayan121 is offline Offline
1,606 posts
since Dec 2006
Jun 1st, 2007
0

Re: Tokenizer with <p id> tags

The file is something like that :

<P ID=1>
CONTENTS
</P>
<P ID=2>
VOLUME I
</P>
<P ID=3>
BOOK FIRST.--A JUST MAN
</P>
<P ID=4>
CHAPTER
I. M. Myriel
II. M. Myriel becomes M. Welcome
III. A Hard Bishopric for a Good Bishop
IV. Works corresponding to Words
V. Monseigneur Bienvenu made his Cassocks last too long
VI. Who guarded his House for him
VII. Cravatte
VIII. Philosophy after Drinking
IX. The Brother as depicted by the Sister
X. The Bishop in the Presence of an Unknown Light
XI. A Restriction
XII. The Solitude of Monseigneur Welcome
XIII. What he believed
XIV. What he thought
</P>
<P ID=5>
BOOK SECOND.--THE FALL
</P>
<P ID=6>
I. The Evening of a Day of Walking
II. Prudence counselled to Wisdom
III. The Heroism of Passive Obedience
IV. Details concerning the Cheese-Dairies of Pontarlier
V. Tranquillity
VI. Jean Valjean
VII. The Interior of Despair
VIII. Billows and Shadows
IX. New Troubles
X. The Man aroused
XI. What he does
XII. The Bishop works
XIII. Little Gervais
</P>
Attached Files
File Type: txt input.txt (21.5 KB, 8 views)
Last edited by katerinaaa; Jun 1st, 2007 at 3:26 am.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
katerinaaa is offline Offline
11 posts
since May 2007

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in C++ Forum Timeline: [c++]-sms like an mobile phone
Next Thread in C++ Forum Timeline: The COM Ports are not communicating properly..





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC