Member Avatar for mini programmer

Hello,

I have a number of long text files, they took from windows system with specific encoding (windows-1256).

Can I count the all different number of words in theses files using Ubuntu commands?

For example: If I have three text files contain respectively

-Hi, hello.
-Hello all.
-Good people in all time.

The output should be: 7.

Thank you

Edited by mini programmer
  • 2 Contributors
  • 5 Replies
  • 360 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by mini programmer

Dani AI

Generated

Brief diagnosis and a compact, reliable fix (addresses both ’s encoding issue and ’s punctuation/capitalization point)

The problem is twofold: the files are encoded in Windows-1256 (so byte-wise tools like the original pipeline see wrong characters) and the tokenization used so far is ASCII-only (tr -cd 'a-zA-Z' strips non‑Latin letters and removes punctuation handling). Changing the terminal encoding only affects display; the file bytes still need to be interpreted with the correct codec or converted.

Step 1 — confirm/convert encoding
Detect a file’s encoding (example):

file -i path/to/file.txt

A reliable conversion to UTF‑8 (one file example):

iconv -f WINDOWS-1256 -t UTF-8 input.txt > input.utf8.txt

Step 2 — Unicode‑aware tokenization and unique counting
A small Python 3 script handles Windows‑1256 input, normalizes Unicode (removes diacritics), treats letters from any script as words, and counts uniques without creating a temporary output file:

#!/usr/bin/env python3
# count_unique.py
import sys, re, unicodedata

pat = re.compile(r'\w+', flags=re.UNICODE)
seen = set()

def norm(w):
    w = unicodedata.normalize('NFKC', w)
    w = ''.join(ch for ch in unicodedata.normalize('NFKD', w)
                if not unicodedata.category(ch).startswith('M'))
    return w.casefold()

for fn in sys.argv[1:]:
    with open(fn, encoding='cp1256', errors='ignore') as f:
        for line in f:
            for m in pat.findall(line):
                t = norm(m)
                if t and not t.isdigit():
                    seen.add(t)

print(len(seen))

Notes and alternatives

  • The script streams files and avoids loading entire files at once, but it does keep the set of unique tokens in memory; a few hundred thousand unique tokens typically requires tens to a few hundred MB of RAM depending on average token length and Python overhead.
  • For very large unique sets, prefer a disk-backed dedupe: convert to UTF‑8, write one token per line, then use the system sort (which can spill to disk) to deduplicate and count unique lines, or insert tokens into an SQLite table with a UNIQUE key and count rows.
  • The example given in the thread should yield 7 unique words when tokenization ignores punctuation and is case‑insensitive; earlier pipelines produced 9 because punctuation/capitalization were not normalized.

Recommended Answers

All 5 Replies

Member Avatar for mini programmer

Thank you readers :)

Now, I found a solution for part of the problem:

cat ~/folderName/* | tr ' ' '\n' | sort | uniq | wc -w

This command calculates the number of unique words in all files in folderName.
It works with English files correctly.

But until now, it does not work with files have windows-1256 encoding.
The terminal can not read the number of words, it prints (0).

I changed encoding of the terminal by:

gconftool --set --type=string /apps/gnome-terminal/profiles/Default/encoding WINDOWS-1256

or I tried to use CP1256 insted of WINDOWS-1256. But until now it does not work :(

Member Avatar for L7Sqr

For your original example, your script gives the answer 9, not 7 as you requested. The command you show does not handle punctuation and capitalization.

You probably want to add some sed-like functionality to remove punctuation and address capitalization. After that you can consider the encoding.

Member Avatar for mini programmer

Thank you.
Are You mean adding like this tr -cd 'a-zA-Z''\n'to the command:

cat ~/folderName/* | tr ' ' '\n'| tr -cd 'a-zA-Z''\n' | sort | uniq | wc -w

Thank you. I added that now, and the encoding is still.

Member Avatar for mini programmer

Sorry, I can not edit my previous post.

Can I find the count of words without needing to use wc command?
For example like that:

tr -cs 'a-zA-Z''\n' '\n'|  sort | uniq  < ~/folderName/*

But I find the error: ~/folderName/: ambiguous redirect
and if I put the folder name between "~/folderName/*" or '~/folderName/*' I found the error which is: No such file or directory

Edited by mini programmer
Member Avatar for mini programmer

Finally, I got this correct solution :)

cat ~/folderName/* | tr ' ' '\n' |tr -cd 'a-zA-Z''\n'| awk '{a[$1]} END {for(i in a) print i}'  > ~/out.txt 
awk 'END {print NR}' ~/out.txt

I hope this help for anyone has same my problem :)

But this command need to a file to save the output (lines/words), Is there any problem if this file contains huge number of lines, e.g more than 500,000 lines?
So, can I combine between these command to remove using this file. Until now I can not find the solution.

Can anyone help me!

Thank you.

Edited by mini programmer
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.