Member Avatar for Konstantinos_1

Ι have many text files with total space 4 GB. In Russian, Greek and English languages.

Is there a way - program - software to find the most common - frequent words in these files?

I want it to produce a list ordered from most to least used words.

I know only C and Matlab. Thanks in advance.

  • 3 Contributors
  • 2 Replies
  • 183 Views
  • 17 Hours Discussion Span
  • Latest Post Latest Post by KushMishra

Dani AI

Generated

Two things matter most for a 4 GB multilingual corpus: correct Unicode handling (Greek + Cyrillic + Latin) and a streaming approach so you never try to load everything into memory at once.

A pragmatic C/C++ solution

  • Normalize file encodings first (prefer UTF‑8 or UTF‑16). On Windows convert input bytes to wide characters (MultiByteToWideChar) so you can use iswalpha/towlower for tokenization across scripts.
  • Use an in‑memory hash table for counts (prefer std::unordered_map<std::wstring,size_t>). Call reserve with a rough expected vocabulary size to avoid rehashing, and use emplace/move to avoid copies.
  • If memory is tight, process files in chunks: build counts for each chunk, write a sorted temporary file of word\tcount, and then perform a k‑way merge of those temp files to produce global counts. After global counts are ready, produce the final frequency ordering by either sorting all pairs or using a min‑heap to keep only the top N.

Example sketch (token loop only):

std::unordered_map<std::wstring,size_t> freq;
void process_wstring(const std::wstring &w) {
  std::wstring cur;
  for (wchar_t ch : w) {
    if (iswalpha(ch)) cur.push_back(towlower(ch));
    else if (!cur.empty()) { ++freq[cur]; cur.clear(); }
  }
  if (!cur.empty()) ++freq[cur];
}

Notes on other suggestions

  • As pointed out, C++ map is simple, but unordered_map will usually be much faster for counting; use map only if you need ordered keys.
  • A web “crawler” (per ) isn’t needed for local files — that term applies to fetching remote pages.

Matlab option

  • Don’t use fileread on 4 GB. Read with fopen/fread in blocks and either use containers.Map (slow for huge vocabularies) or call a compiled C program / MEX to do the heavy lifting.

Final tips

  • Detect file encoding (BOM or sample bytes), normalize, strip punctuation consistently, and decide if you want simple token forms or language‑aware stemming (the latter requires extra libraries). Compile 64‑bit for large memory and use chunking + merge if RAM is limited.

Recommended Answers

All 2 Replies

Member Avatar for meta.quota

to reduce code lines, i'd suggest You use C++ <map> .. it's much more easier .. you just need a helper function named, maybe (split) which is a vector of string that takes as an argument a const string reference ...

get the sample code here

You just have to add your file handlers so instead of the above program getting input from the keyboard, it gets it from the specified file ...

******* HAVE PHUN C0DiNG *******

commented: Good advice +14
Member Avatar for KushMishra

Hi, I am not sure but probably a software called "Crawler" may fix your issue.
This crawler is used by many websites also like Google etc.
May be it would be of some help to you as well.

commented: nice attempt to solve the question +0
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.