Word Frequency using Python

bumsfeld 2 2K Views

This program uses Python module re for splitting a text file into words and removing some common punctuation marks. The word:frequency dictionary is then formed using try/except. In honor of 4th of July the text analyzed is National Anthem of USA (found via Google).

# another word frequency program, uses re
# tested with Python2.4.3   HAB

import re

# this one in honor of 4th July, or pick text file you have!!!!!!!
filename = 'NationalAnthemUSA.txt'

# create list of lower case words, \s+ --> match any whitespace(s)
# you can replace file(filename).read() with given string
word_list = re.split('\s+', file(filename).read().lower())
print 'Words in text:', len(word_list)

# create dictionary of word:frequency pairs
freq_dic = {}
# punctuation marks to be removed
punctuation = re.compile(r'[.?!,":;]') 
for word in word_list:
    # remove punctuation marks
    word = punctuation.sub("", word)
    # form dictionary
    try: 
        freq_dic[word] += 1
    except: 
        freq_dic[word] = 1
    

print 'Unique words:', len(freq_dic)

# create list of (key, val) tuple pairs
freq_list = freq_dic.items()
# sort by key or word
freq_list.sort()
# display result
for word, freq in freq_list:
    print word, freq

Dani AI

Generated

The original snippet from is a nice, simple Python-2 demo, but a few practical improvements make the result more robust for real text files (Unicode quotes, contractions, stopwords, and modern Python). 's question about descending order is best solved with a counter that already yields most-common terms; 's point about simplifying the incrementing logic is also on target. Below is a compact Python‑3 approach that keeps counts, handles typographic apostrophes, optionally expands a few common contractions, and supports a stopword file.

# Python 3: Unicode-safe word frequency, optional stopwords, contraction expansion
import re
from collections import Counter

def normalize(text):
    text = text.replace("’", "'").lower()                     # normalize curly quotes
    contractions = {"it's": "it is", "there's": "there is"}    # extend as needed
    pat = re.compile(r"\b(" + "|".join(map(re.escape, contractions)) + r")\b")
    return pat.sub(lambda m: contractions[m.group(0)], text)

with open("NationalAnthemUSA.txt", encoding="utf-8") as f:
    txt = normalize(f.read())

words = re.findall(r"\b[a-z]+(?:'[a-z]+)?\b", txt)             # keeps contractions like "there's"
try:
    with open("stopwords.txt", encoding="utf-8") as s:
        stop = set(w.strip() for w in s)
except FileNotFoundError:
    stop = set()

counts = Counter(w for w in words if w not in stop)
for word, freq in counts.most_common(50):
    print(word, freq)

Troubleshooting notes: if punctuation like commas still appear, check for non-ASCII punctuation (smart quotes, em‑dashes) and normalize them first. To treat "there" and "there's" as the same concept, either expand contractions (shown) or use a lemmatizer/tokenizer (NLTK/spaCy) for deeper normalization. For large corpora, use Counter and most_common() (descending order) to avoid manual sorting.

Member Avatar for kenmeck03
kenmeck03 0 Newbie Poster

How would you take this and organize the words that appear in descending order not alphabetical.

Member Avatar for bumsfeld
bumsfeld 413 Nearly a Posting Virtuoso

Do you mean highest frequency first?

Simply add this to the end of the code:

print '-'*30

print "sorted by highest frequency first:"
# create list of (val, key) tuple pairs
freq_list2 = [(val, key) for key, val in freq_dic.items()]
# sort by val or frequency
freq_list2.sort(reverse=True)
# display result
for freq, word in freq_list2:
    print word, freq
Edited by bumsfeld because: n/a
Member Avatar for bipratikgoswami
bipratikgoswami 0 Newbie Poster

this code is useful...

Member Avatar for Member #562630
Member #562630

nice piece of code :) useful indeed

Member Avatar for mattp23
mattp23 0 Newbie Poster

line 21 onwards:

# form dictionary
    try:
         freq_dic[word] += 1
    except:
         freq_dic[word] = 1

Could be replaced by:

freq_dic[word] = freq_dic.get(word,0) + 1

gets rid of the try except and just makes things a little neater.

Member Avatar for nawaf_ali
nawaf_ali 0 Newbie Poster

what can I add to this code to remove some words listed in some other file prior doing the frequency listing?

Member Avatar for luisbeta04
luisbeta04 0 Newbie Poster

This doesn't seem to remove any punctuation marks from the text file, and reads 'it' separately from 'it,'.

What might the problem be?

Member Avatar for sujit.shakya.3
sujit.shakya.3 0 Newbie Poster

But, does it works for "There" and "There's".

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

Edit Preview