Member Avatar for Shantanu88d

i have been given an assignment, to find frequencies of all words in a large text file. I have tried a program which finds the same in a sample string. Done by taking that string in an array. But in case of a text file spanning many pages with thousands of words, won't that array eat up a lot space? I have been asked to consider performance as a prime criteria.
Any suggestion will be awesome

  • 5 Contributors
  • 10 Replies
  • 593 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by edwinhermann

Dani AI

Generated

The OP's memory concern is valid: in-memory arrays can blow up on multi‑page texts. This thread already shows a character-stream idea (), a regex+in‑memory count (), and a suggestion to use a database (). The practical choices for large inputs are (a) stream + disk-backed counting, (b) external sort + aggregate, or (c) approximate sketches when exact counts are not required. Each trades RAM for I/O or accuracy.

A robust, practical pipeline: read the file in streaming fashion, perform Unicode normalization and language-aware tokenization, keep an in-memory accumulator for a moderate number of unique tokens, and flush batches to a disk-backed store (SQLite, LevelDB, LMDB) inside a transaction. This keeps hot counts fast while avoiding a full in‑RAM hash table. Example (Python outline):

import sqlite3, unicodedata, regex as re
from collections import Counter

conn = sqlite3.connect('counts.db')
conn.execute('CREATE TABLE IF NOT EXISTS counts(word TEXT PRIMARY KEY, cnt INTEGER)')
tok = re.compile(r'\p{L}+')
batch = Counter()
THRESH = 100000

with open('large.txt', encoding='utf-8') as fh:
    for line in fh:
        s = unicodedata.normalize('NFC', line)
        for w in tok.findall(s):
            batch[w.lower()] += 1
        if len(batch) > THRESH:
            with conn:
                for w,c in batch.items():
                    conn.execute('INSERT INTO counts(word,cnt) VALUES(?,?) ON CONFLICT(word) DO UPDATE SET cnt=cnt+?',
                                 (w,c,c))
            batch.clear()
if batch:
    # same flush as above
    pass
conn.close()

For a zero‑programming alternative that scales well on Unix systems, stream tokens into a file and use external sort + aggregation (sort | uniq -c). For extremely large corpora where exact counts are infeasible, consider Count‑Min Sketch or Space‑Saving algorithms to estimate heavy hitters.

For UTF‑8/Arabic: always open with UTF‑8, normalize (NFC), use a Unicode‑aware tokenizer (ICU or libraries that support \p{L}), and consider stripping diacritics or normalising alef/tashkeel variants if language rules require it. Batch DB writes and tuned transactions dramatically improve performance; tuning durability settings can speed bulk runs but must be used with caution.

Recommended Answers

All 10 Replies

Member Avatar for edwinhermann

I think this would work reasonably fast:

<?php
$filename = "/path/to/file.txt";
$handle = fopen($filename,"r");
if ($handle === false) {
  exit;
  }
$word = "";
while (false !== ($letter = fgetc($handle))) {
  if ($letter == ' ') {
    $results[$word]++;
    $word = "";
    }
  else {
    $word .= $letter;
    }
}
fclose($handle);
print_r($results);
?>

Note: This assumes the file is in the format <word><space><word><space><word>.... etc.

Edited by edwinhermann because: n/a
Member Avatar for pritaeas

You can always put your results in a mysql table, if your worried that memory will become a problem.

Member Avatar for Member #120589

You came here for a suggestion and got a complete solution. Lucky guy.

Member Avatar for edwinhermann

You came here for a suggestion and got a complete solution. Lucky guy.

I just re-read the original post, and now I realise it was for an assignment. I suppose I shouldn't have provided the complete solution. Oh well :)

Member Avatar for Member #120589

Haha, he'll have to noobie-fy it it he wants to pass it off as his own though! ;)

Member Avatar for Member #120589

You could also do this:

$str = file_get_contents("text.txt"); //get string from file - no error handling here though
preg_match_all("/\b(\w+[-]\w+)|(\w+)\b/",$str,$r); //place words into array $r - this includes hyphenated words
$c = array_count_values(array_map("strtolower",$r[0])); //create new array - with case-insensitive count 
foreach($c as $key => $val){
	echo $key . " [" . $val . "]<br />";  //output data	
}

This will give you output for ASCII character words. However multibyte characters (â etc) won't work. That needs something more sophisticated.

I bet somebody could write a better regex than I've used though.

Edited by Member #120589 because: n/a
Member Avatar for Shantanu88d

You came here for a suggestion and got a complete solution. Lucky guy.

what can i say, edwinhermann is really kind !!

Member Avatar for Shantanu88d

edwinhermann and ardav...thanks for your suggestions and answers !! :)

Member Avatar for khan12345

i have a text file written in utf-8 format like arabic script. for that suggest me what to do. how to output text in same arabic script

Member Avatar for edwinhermann

Khan - please start a new thread. This thread is solved and it's not appropriate to post additional messages here.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.