RSS Forums RSS

Determining the number of unique words in a .txt file

Please support our C++ advertiser: Programming Forums
Reply
Posts: 13
Reputation: matt_570 is an unknown quantity at this point 
Solved Threads: 0
matt_570 matt_570 is offline Offline
Newbie Poster

Determining the number of unique words in a .txt file

  #1  
Dec 2nd, 2008
Hey I have to write a program that reads a text file that contains a list of words, 1 word per line. I have to store the unique words and count the occurrences of each unique word. When the file is completely read, I have to print the words and the number of occurrences to a text file. The output should be the words in alphabetical order along with the number of times they occur. Then print to the file some statistics:

I have to use character arrays instead of strings.
I must use the linear search (something that looks like this)
array.int search ( int array[], int number, int key )
{
	int pos = 0;
	
	while(pos < number && array[pos] != key)
		pos++;
	
	if( pos == number)
		pos = 0;
	else
		pos = 1;
	
	return pos;
}	
to determine if a word is in the array. The array is an array of structures and that the key is a char array so the string comparison must be used. The search task should be a separate function.
The search must be a separate function that returns an integer values. I cant use a for loop and the function must have only one return statement.
I tried to start off by reading the files and storing the words
#include <iostream>
#include <fstream>
#include <cstdlib>
#include <string>
#include <iomanip>
using namespace std;

void displayFile( char []);

void main ()
{
	int const wordLength = 21;
	int const Num = 101;
	int const fileSize = 255;
	
	
	char filename[fileSize];
	ifstream inFile;
	
	
	struct
	{
		char word[wordLength];
		int count;
	} wordCount;
	
	wordCount array[Num];
	
    cout << "Please enter the name of thr file you wish to open: "<< endl;
	cin >> filename;
	
	displayFile (filename);
	
}

void displayFile (char fileName [] )
{
    ifstream infile;
	char array [101];
    char line [101];
	int ch;
    int i;
	infile.open (fileName);

    while ((ch = infile.peek()) != EOF) 
    {
        infile.getline (line, 101);
        line = array[i];
        i++;
    }
	
   for (int j = 0; j<101; j++)
	cout << array[j] << endl;
    infile.close ();

Any tips would be appreciated, and thanks in advance.
AddThis Social Bookmark Button
Reply With Quote  
Posts: 13,865
Reputation: Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute Ancient Dragon has a reputation beyond repute 
Solved Threads: 1230
Moderator
Featured Poster
Ancient Dragon's Avatar
Ancient Dragon Ancient Dragon is online now Online
Most Valuable Poster

Re: Determining the number of unique words in a .txt file

  #2  
Dec 2nd, 2008
I would first create a structure
struct words
{
    std::string word;
    int count;
};

Next an vector (array) of those structures
vector<words> wordArray;

Now, then the program reads a line, check wordArray if its already in the array. if it is then increment the count. If not, then add a new element to the vector. When done, you will have the information you need to display on the screen.

Another method is to use a <map>, but implementation is left for someone else.
Reply With Quote  
Posts: 110
Reputation: Laiq Ahmed is on a distinguished road 
Solved Threads: 13
Laiq Ahmed Laiq Ahmed is offline Offline
Junior Poster

Re: Determining the number of unique words in a .txt file

  #3  
Dec 3rd, 2008
I think you should look at the below code more carefully and must understand the complexity and map datastructure,

if you've any confusion post, I will explain you in more detail then.
  1. fstream inFile("C:\\Laiq.txt");
  2.  
  3. string line;
  4. std::map<string, int> mapWordCount;
  5. string word;
  6. std::string::size_type endWord;
  7. std::string::size_type startWord;
  8. while (std::getline (inFile,line)) {
  9. startWord = 0;
  10. while ((endWord = line.find_first_of(" ", startWord))!= line.npos)
  11. {
  12. word = line.substr (startWord, endWord - startWord);
  13. mapWordCount[word]++;
  14. startWord = endWord+1;
  15. }
  16. }
  17. inFile.close();
  18.  
  19.  
  20. for(map<string,int>::const_iterator iter = mapWordCount.begin();
  21. iter != mapWordCount.end(); ++iter) {
  22. cout<< "The Frequency of word: "<< iter->first << " is : "<< iter->second <<endl;
  23. }
Last edited by Laiq Ahmed : Dec 3rd, 2008 at 4:06 am. Reason: Changing Code Intendation
Reply With Quote  
Posts: 2,000
Reputation: ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of 
Solved Threads: 331
ArkM's Avatar
ArkM ArkM is offline Offline
Postaholic

Re: Determining the number of unique words in a .txt file

  #4  
Dec 3rd, 2008
You have two different problems:
1. Tokenize the input stream (extract words from the stream).
2. Build word dictionary.
The second one has a very simple solution: use std::<map> data structure as Laiq Ahmed mentioned above. However you need another code to process every word with the map-based word dictionary:
if the next word is found in the map then
    do notning or increment this word counter
else
    insert new word in the map
endif
There are lots of methods to solve the 1st problem. For example:
open file stream
create an empty map
loop // until eof
   skip non-letters
   clear word buffer // use std::string::clear()
   append letters to the word buffer
   process the word with the map // see #2
endloop
process a possible last word
traverse map (file word dictionary)
Summary:
- use std::ifstream
- use std::string for word buffer
- use std::map<std::string,int> for dictionary with counters
You have a good chance to write a simple and clear code after a proper functional decomposition of pseudocode snippets ...
Reply With Quote  
Posts: 1,572
Reputation: Lerner is a name known to all Lerner is a name known to all Lerner is a name known to all Lerner is a name known to all Lerner is a name known to all Lerner is a name known to all 
Solved Threads: 239
Lerner Lerner is offline Offline
Posting Virtuoso

Re: Determining the number of unique words in a .txt file

  #5  
Dec 3rd, 2008
>>I have to use character arrays instead of strings.
I must use the linear search

That pretty much precludes use of a map. It also means you can't use STL strings within a struct, and may preclude use of a vector to hold the structs, too.

You could use a C style string with either static or dynamic memory within the struct. The static version will limit the maximum length of any possible string entered, but since these strings will be words, then it would be unlikely that any given word would have more than 20 or 30 letters per word. Using dynamic memory to store the string within the struct will minimize the memory wasted by using strings of length less than than maximum length in the static version, but will require you to manage your own memory, which is a bit of a hassle, though not an overwhelming task by any means.

The non-STL structures you could use to store the structs could be either an array or a list. If you don't know the maximum number of possible unique words you could encounter then using a list might be advantageous. You could make a guess as to the max number of words, but that isn't quite as predictable as the maximum size of each word. Linear searching is possible with either lists or arrays.

Sorting the structures by string in alphabetical order before printing/sending values to file could be done in either of several different ways. For beginners, bubble sorts with arrays and insertion sorts with lists seem pretty popular.
Klatu Barada Nikto
Reply With Quote  
Posts: 2,000
Reputation: ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of 
Solved Threads: 331
ArkM's Avatar
ArkM ArkM is offline Offline
Postaholic

Re: Determining the number of unique words in a .txt file

  #6  
Dec 3rd, 2008
Sorry, I have not noticed those absurd restruictions.
The only result of this idiotic methodology: a pupil sheds tears on DaniWeb then use copy/paste for extremelly ineffective codes in early 60-th style (or worse)...
Reply With Quote  
Posts: 13
Reputation: matt_570 is an unknown quantity at this point 
Solved Threads: 0
matt_570 matt_570 is offline Offline
Newbie Poster

Re: Determining the number of unique words in a .txt file

  #7  
Dec 3rd, 2008
The maximum length of a word is 20, so the character array needs to be 21. The maximum numbers of words is 100, if there are more than 100 words I need to return a message saying something like "The stats did not use the words after the 100th word"

Anyways, I'm having a hard time even starting the program. I'm starting off by trying to read and store the words of the text file. My text book has code for reading line by line a text file and then displaying it. So I figured I should start with that. Heres the code-

#include <iostream>
#include <fstream>
#include <cstdlib>   // needed for exit()
#include <string>
using namespace std;

int main()
{
  string filename = "text.dat";  // put the filename up front
  string line;
  ifstream inFile;
  
  inFile.open(filename.c_str());

  if (inFile.fail())  // check for successful open
  {
    cout << "\nThe file was not successfully opened"
	 << "\n Please check that the file currently exists."
	 << endl;
    exit(1);
  }

  // read and display the file's contents
  while (getline(inFile,line))
    cout << line << endl;

  inFile.close(); 

  cin.ignore();  // this line is optional

  return 0;
}

Heres my code split into two functions, however it does not compile. I get an error at the while statement.-
#include <iostream>
#include <fstream>
#include <cstdlib>
#include <string>
#include <iomanip>
using namespace std;

void displayFile( char []);

void main ()
{
	int const wordLength = 21;
	int const Num = 101;
	int const fileSize = 255;
	char filename[fileSize];
	
	cout << "Please enter the name of the file you wish to open: "<< endl;
	cin.getline(filename,fileSize);
	
	displayFile (filename);
	cin.ignore();
}

void displayFile (char fileName[] )
{
    ifstream inFile;
	
    char line [101];
	
	inFile.open(fileName);

 while (getline(inFile, line))
    cout << line << endl;

	inFile.close(); 

	return 0;
}
My code is nearly identical except for the fact that I use character arrays instead of strings, so I'm not sure why its not compiling.
Also, is this a good way to start off the program? Or should I try something else.
Reply With Quote  
Posts: 13
Reputation: matt_570 is an unknown quantity at this point 
Solved Threads: 0
matt_570 matt_570 is offline Offline
Newbie Poster

Re: Determining the number of unique words in a .txt file

  #8  
Dec 3rd, 2008
Ok I figured out I need to use "inFile.getline(line,length)" instead of "getline(inFile, line)". I created a struct like stated early. However I'm having trouble storing the words from the text file.

I use this function-

void displayFile (char fileName[], words array[] )
{
	int i = 0;
    ifstream inFile;
	
    char line [101];
	
	inFile.open(fileName);

 while (inFile.getline(line,101))
 {   
	cout << line << endl;
    array[i].word = line;
    i++;
 }   
	inFile.close(); 

}
Where array, is an array of structs made up of character array"word" and an integer "count". Any help on storing the words and the number of words in the array would be helpfull.
Reply With Quote  
Posts: 2,000
Reputation: ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of ArkM has much to be proud of 
Solved Threads: 331
ArkM's Avatar
ArkM ArkM is offline Offline
Postaholic

Re: Determining the number of unique words in a .txt file

  #9  
Dec 4th, 2008
Read-a-line-in-a-char-array solution has an obvious defect: you must define max line length a priori.
Look at the function which can get words from an input stream (not only from fstream) directly:
  1. // #include <cctype> dependency
  2. typedef unsigned char Uchar; // needed for isalpha()
  3. /// Get or skip the next word from a stream.
  4. /// wbufsize is the max word length + 1 (for null char)
  5. bool getWord(char* word, int wbufsize, std::istream& is)
  6. {
  7. int i = 0, n = wbufsize - 1; // max letters
  8. bool fill = (word && n > 0); // word wanted
  9. char ch; // current char
  10.  
  11. while (is.get(ch) && !isalpha(Uchar(ch)))
  12. ; // skip delimiters
  13. if (is) // have a letter
  14. do {
  15. if (fill) { // have a room
  16. word[i] = ch; // append letter
  17. if (i >= n) // no more room
  18. fill = false;
  19. }
  20. ++i; // letters counter
  21. } while (is.get(ch) && isalpha(Uchar(ch)));
  22. if (word)// have a buffer
  23. word[i] = '\0'; // end of word
  24. return i > 0; // have a word
  25. }
Of course, it's possible to adopt this algorithm to extract words from a char array.
Reply With Quote  
Posts: 110
Reputation: Laiq Ahmed is on a distinguished road 
Solved Threads: 13
Laiq Ahmed Laiq Ahmed is offline Offline
Junior Poster

Re: Determining the number of unique words in a .txt file

  #10  
Dec 4th, 2008
I tried a different Approach to achieve the same
  1. char* arr = " Hallo WOrld";
  2. char* ptrFirst = arr;
  3. char* ptrIter = arr;
  4. int nWordCount = 0;
  5.  
  6. // Base Condition.
  7. while (isspace (*ptrFirst))
  8. ++ptrFirst;
  9.  
  10. ptrIter = ptrFirst;
  11.  
  12. while (ptrFirst) {
  13.  
  14. while (isalpha(*ptrIter)) {
  15. ++ptrIter;
  16. }
  17.  
  18. ++nWordCount;
  19.  
  20. while (isspace (*ptrIter))
  21. ++ptrIter;
  22.  
  23. if (ptrFirst == ptrIter)
  24. break;
  25.  
  26. ptrFirst = ptrIter;
  27. }
  28.  
  29. cout << nWordCount-1 <<endl;
  30.  

Hope this approach will help you understand?
Reply With Quote  
Reply

Only community members can participate in forum threads. You must register or log in to contribute.



Other Threads in the C++ Forum
Views: 2005 | Replies: 25 | Currently Viewing: 1 (0 members and 1 guests)

 

Thread Tools Display Modes
Forums | Blogs | Tutorials | Code Snippets | Whitepapers | RSS Feeds | Advertising
All times are GMT -4. The time now is 3:11 pm.
Newsletter Archive - Sitemap - Privacy Statement - Acceptable Use Policy - Contact Us
Forum system based on vBulletin Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
©2003 - 2008 DaniWeb® LLC