Search DaniWeb - tokenize

Tokenize 17 Years Ago by torbecire … having a problem with tokenizing "wddw". I can tokenize if they is a sign like (,). How can I… tokenize this without getting an error. I have tried charAt(0), … Tokenize 16 Years Ago by letlet_pogs hello guyz,,who knows how to tokenize???i mean,,i am about to start to make "String Calculator" but i don't know how to tokenize,,,do you know any sites that could explain best about tokens??thank you ahead,,,, tokenize an input text file.. 17 Years Ago by minamoda … 1999[/QUOTE] i wrote a program and the part for tokenize is: [code] line= BbmReader.getFileReader("input.txt").readLine… Re: tokenize an input text file.. 17 Years Ago by minamoda "BbmReader" is a class in a library which we use in our school i have to use it and i think there isnt any problem with it but my program doesnt tokenize all the text file. only the first line. And by the way, I HAVE TO use tokenizer because our teacher wants it. Thank you so much for the help but still it doesnt do what i want. :( tokenize text input 15 Years Ago by doomfrawen … tab with 2 RTB , my problem is how do I tokenize or read the input line by line, i guess that… Tokenize text from input file 15 Years Ago by HoboConductor … for reading a text file, and I've tried to tokenize the words, however I have come into many syntax errors… Re: Tokenize 17 Years Ago by Samudricka i think it is better to use character array to store each character and get it....sample syntax is like this.....[CODE] char[] i=new char[10]; String u = (String) stoke.nextToken(); i= u.toCharArray();[/CODE] So that you can get as each character....... Re: Tokenize 16 Years Ago by Alex Edwards How do you need your tokens to be parsed? An example... [code=java] public class TokenMaster{ public static void main(String... args){ String myReallyBIGString = "ABCDEFGHIJKLMNOPQRSTUVWXYZ\n" + "ALPHABETICAL!\n" + "SOMETIMES I FEEL I'VE GOT TO 'AH-AH' RUN AWAY, I'VE GOT TO... " + &… Re: Tokenize 16 Years Ago by letlet_pogs thank you very much,,,may i as,,how useful are tokens??? Re: Tokenize 16 Years Ago by yilmazhuseyin you can also use [URL="http://java.sun.com/javase/6/docs/api/java/util/StringTokenizer.html"]String Tokenizer[/URL] class. Re: Tokenize 16 Years Ago by Alex Edwards [QUOTE=yilmazhuseyin;683536]you can also use [URL="http://java.sun.com/javase/6/docs/api/java/util/StringTokenizer.html"]String Tokenizer[/URL] class.[/QUOTE] Or not, because StringTokenizer is considered an Obsolete class that receives no further support in future releases. Furthermore, String.split() is better because it takes … Re: Tokenize 16 Years Ago by digitaldesperad You may use the special PHP function strtok <?php $myText="online pharmacy web solution"; $strToken=strtok($myText," "); while($strToken){ echo $strToken."<br>"; $strToken=strtok(" "); } ?> this my hel;p u Re: Tokenize 16 Years Ago by Ezzaral [QUOTE=digitaldesperad;698254] this my hel;p u[/QUOTE] Sure, maybe... if he was using PHP, but since this is the Java forum and his code is Java, your suggestion seems to fall a little bit short of helpful. And this isn't a chat room, so you can leave the IM-speak at the door. Is there a way to tokenize an array of strings 19 Years Ago by Halcyon … timer here so bear with me. I'm trying to tokenize an array of strings. This program reads an array of… correct) if there is nothing else to tokenize(two name string), dont tokenize it tokenize the rest of the elements(3 name string… Parse or Tokenize String 17 Years Ago by Akilah712 …lt;string> sets; //get the sets - name+sequence Tokenize (contents, sets, ">"); //stores the splitted … (int x = 0; x < sets.size(); x++){ Tokenize (sets[x], dna, "\n"); } //store the …dna[j]); j = j + 2; } }//End processFile void Tokenize(const string& str,vector<string>& tokens… Re: tokenize an input text file.. 17 Years Ago by Ezzaral You need to have the read in a loop like so [code]while ((line = reader.readLine()) != null){ StringTokenizer tokenizer = new StringTokenizer(line," "); // do stuff }[/code]I have no idea where the "BbmReader" class comes from, so I can't speak to any specifics on that, but the general loop should work the same. By the … Re: tokenize an input text file.. 17 Years Ago by Ezzaral It certainly will if you use the loop as I indicated. Get the reader one time and then use that loop to read lines from it. Re: tokenize an input text file.. 17 Years Ago by minamoda You tell me that [code] while ((line = reader.readLine()) != null){ StringTokenizer tokenizer = new StringTokenizer(line," "); // do stuff }while ((line = reader.readLine()) != null){ StringTokenizer tokenizer = new StringTokenizer(line," "); // do stuff } [/code] works i am confused now if "… Re: tokenize an input text file.. 17 Years Ago by Ezzaral Of course it's endless - because you are not ever reading the next line. The loop that I wrote reads the line into "line" each iteration of the loop and checks it against null. Yours calls readLine() [U]one[/U] time and keeps looping while "line" is not null - so the loop is either endless or never executes in the first place. … Re: tokenize text input 15 Years Ago by sknake This should get you started: [code=vb.net] Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click Dim txtString As String() = System.IO.File.ReadAllLines("C:\file.txt") Dim studentList As New System.Collections.Generic.Dictionary(Of Integer, Decimal()) For Each line As … Re: tokenize text input 15 Years Ago by doomfrawen Thank you so much for your help, that really helped me, it was pretty easy to understand your code, now i can implement it into my program thanks a lot :D:D Re: tokenize text input 15 Years Ago by sknake You're welcome Please mark this thread as solved if you have received an answer to your question and good luck! Re: Tokenize text from input file 15 Years Ago by Fbody For starters, your main() is declared with a [B]void[/B] return. It should ALWAYS be declared with an [B]int[/B] return. Beyond that, we can't really help you without specific error information. Re: Tokenize text from input file 15 Years Ago by caut_baia you can use [code] strtok (char*,char); [/code] something like this: [code] char* str="string literal"; char* string=strtok(str," "); char* literal=strtok(NULL," "); [/code] Re: how to tokenize a long string by using another string 18 Years Ago by rpjanaka public void tokenize(){ StringTokenizer tokenizer = new StringTokenizer("{ok,[{operators,'sri','mob',45,… Tokenize a line with strtok & reverse line out.. 15 Years Ago by MAV_DevWantaB :-/AnyWay,, Here is what it is supposed to happen.... [COLOR="Green"]The input is a char array initialized with line of Text,, the program should tokenizes the line with the "strtok" function and should output the line of text in reverse order.... [/COLOR] I need to set up a "strcpy"..... I'm almost there just … Re: Tokenize a line with strtok & reverse line out.. 15 Years Ago by Narue [B]>strcpy( *revrseStrng, string );[/B] I don't know what you imagined this to do, but there's no magic in C. If you want to build a string from the tokens of another, you do it just as one would expect: [code] while more tokens get token append token to string print string [/code] Likewise, to build the same string in reverse you … Re: Tokenize a line with strtok & reverse line out.. 15 Years Ago by MAV_DevWantaB Well almost gooter done,,,, [COLOR="Red"][I]just need to run through the token[i] array in reverse order to get out put to look like example...[/I][/COLOR] Per your examples this is what the output should be: [B]"hello world" -> "dlrow olleh"[/B] I have gotten a bit further with the help of some kind souls… Re: Tokenize a line with strtok & reverse line out.. 15 Years Ago by Narue [B]>Per your examples this is what the output should be: >"hello world" -> "dlrow olleh"[/B] Um, no? Watch again: [code] 1) Reverse and add "hello": "olleh" 2) Reverse and add "world": "olleh dlrow" 3) Reverse the whole thing: "world hello" [/code] Re: Tokenize a line with strtok & reverse line out.. 15 Years Ago by MAV_DevWantaB [QUOTE=Narue;1079141][B]>Per your examples this is what the output should be: >"hello world" -> "dlrow olleh"[/B] Um, no? Watch again: [code] 1) Reverse and add "hello": "olleh" 2) Reverse and add "world": "olleh dlrow" 3) Reverse the whole thing: "world hello&…