Member Avatar for otto531

I've been looking around for an answer to this but have had no luck. I need to take two files and print the top most frequent words they have in common as well as their combined(sum) frequencies. This might be simple but I'm pretty new to programming. Any help?

def mostFrequent(word,frequency,n):
   my_list = zip(word,frequency) #combine the two lists
   my_list.sort(key=lambda x:x[1],reverse=True) #sort by freq
   words,freqs = zip(*my_list[:n]) #take the top n entries and split back to seperate lists
   return words, freqs #return our most frequent words in order   

from wordFrequencies import * #gives both the word and its frequency in a file
L1 = wordFrequencies('file1.txt')
words1 = L1[0]
freqs1 = L1[1]
L2 = wordFrequencies('file2.txt')
words2 = L2[0]
freqs2 = L2[1]
print mostFrequent(words,freqs,20)

I've tried

L1 = WordFrequencies('file1.txt')
words1 = set(L1[0])
freqs1 = set(L1[1])
L2 = WordFrequencies('file2.txt')
words2 = set(L2[0])
freqs2 = set(L2[1])
words3 = words1.intersection(words2)
freqs3 = freqs1.intersection(freqs2)
print mostFrequent(words3,freqs3,20)

but it didn't work. It outputed the wrong words

  • 4 Contributors
  • 3 Replies
  • 327 Views
  • 13 Hours Discussion Span
  • Latest Post Latest Post by bumsfeld

Dani AI

Generated

The cleanest approach is to keep a word→count mapping for each file (dict or collections.Counter), take the intersection of keys, then add the counts for only those common words. Converting the frequency list to a set (as in the original attempt) destroys the word↔frequency relationship, which is why the wrong words appeared — a point already noted by . 's idea of building per-file maps is on the right track; the snippet below implements that using a robust tokenizer and Python's Counter.

import re
from collections import Counter

def top_common_words(path1, path2, n=20):
    token = re.compile(r"[A-Za-z0-9']+")
    def iter_words(p):
        with open(p, 'r', encoding='utf-8') as f:
            for line in f:
                for w in token.findall(line.lower()):
                    yield w

    c1, c2 = Counter(iter_words(path1)), Counter(iter_words(path2))
    common = set(c1) & set(c2)
    combined = Counter({w: c1[w] + c2[w] for w in common})
    return sorted(combined.items(), key=lambda t: (-t[1], t[0]))[:n]

# Example: top_common_words('file1.txt', 'file2.txt', 20)

Practical notes: normalize case and choose a tokenizer that fits the text (handle apostrophes, hyphens, numbers as needed). For very large files, stream line-by-line (the example does this) to avoid building huge strings in memory. If stop words (common words like "the", "and") should be ignored, filter them out before counting. Tie-breaking is done above by alphabetical order after frequency.

Troubleshooting: confirm that the function producing words/frequencies returns a mapping (word→count) rather than parallel lists turned into sets; test on tiny sample files where expected counts are known to validate tokenization and normalization before running on real data.

Recommended Answers

All 3 Replies

Member Avatar for Gribouillis

Use dictionaries

D = [dict(zip(*WordsFrequencies(name))) for name in ['file1.txt', 'file2.txt']]
common_words = set(D[0]) & set(D[1])
L = [(w, D[0][w] + D[1][w]) for w in common_words]
# sort by decreasing frequencies, solve ties by increasing alphabetical order.
L.sort(key = lambda t: (-t[1], t[0]))
L = L[:20]
Edited by Gribouillis
Member Avatar for bumsfeld

Sets will be ordered by hash order, so you loose the relationship of word and frequency data.

It could be easier to use collections.Counter() on each of your text files and go from there.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.