Tokenize

[letlet_pogs]

hello guyz,,who knows how to tokenize???i mean,,i am about to start to make "String Calculator" but i don't know how to tokenize,,,do you know any sites that could explain best about tokens??thank you ahead,,,,

[Alex Edwards]

How do you need your tokens to be parsed?

An example...

public class TokenMaster{
	public static void main(String... args){
		String myReallyBIGString = "ABCDEFGHIJKLMNOPQRSTUVWXYZ\n"
					 			 + "ALPHABETICAL!\n"
					 			 + "SOMETIMES I FEEL I'VE GOT TO 'AH-AH' RUN AWAY, I'VE GOT TO... "
					 			 + "'AH-AH' GET AWAY!!!\n"
					 			 + "RING RING RING RING RING RING RING... BANANA PHONE! 'doot, doot doot doot doot doot'"
					 			 + "DING-DONG DING-DONG DING-DONG DING... BANANA PHONNNNNE!!!!!";

		String tokens[] = myReallyBIGString.split("\n"); // each token is determined by the newline character

		for(String element : tokens)
			System.out.println(element); // printing the individual tokens generated from String.split("\n")

		System.out.println("\nTokens counted: " + tokens.length);
	}
}

[letlet_pogs]

thank you very much,,,may i as,,how useful are tokens???

[yilmazhuseyin]

you can also use String Tokenizer class.

[Alex Edwards]

you can also use String Tokenizer class.

Or not, because StringTokenizer is considered an Obsolete class that receives no further support in future releases.

Furthermore, String.split() is better because it takes regular expressions into consideration. StringTokenizer does not.

[digitaldesperad]

You may use the special PHP function strtok

<?php

$myText="online pharmacy web solution";

$strToken=strtok($myText," ");

while($strToken){

echo $strToken."
";

$strToken=strtok(" ");

}

?>
this my hel;p u

[Ezzaral]

this my hel;p u

Sure, maybe... if he was using PHP, but since this is the Java forum and his code is Java, your suggestion seems to fall a little bit short of helpful.

And this isn't a chat room, so you can leave the IM-speak at the door.