Member Avatar for Member #361407

Hi a while back you guys helped me make a good spell checker, now i want to advance it so that if the difference between 2 words is small it will change them, I think i need to use diiflib for it but i don't know how to do itmy code is below, along with an attached dictionary file

dict  = open("DictionaryE.txt", "r").readlines()
test= open("README.txt", "r").read()

print dict[0:5]

correct = []
unique = []
test = test.lower()
list_words = test.split(' ')
l = []


for line in list_words:
	if line in dict:
		correct.append(line)

for line in correct:
    l.append(line.strip())

correct = set(l)
		
bleh = str(correct)
open("cheese.txt", "w").write(bleh)

Dani AI

Generated

A short, practical plan: stop scanning the whole dictionary for every test word, strip punctuation before checking, and do fuzzy-matching only for words that truly miss. Load the dictionary once (keep a set for fast membership tests and a list for fuzzy lookups), normalize to lower case, and tokenize with a regex so punctuation like commas and exclamation marks don't stay attached. Using a set makes membership checks effectively constant time. (docs.python.org)

Example workflow (tokenize → membership → fuzzy only on misses). This keeps the expensive fuzzy call to one per unknown word instead of one per dict-entry per word:

import re
import difflib

with open("DictionaryE.txt", "r", encoding="utf-8") as f:
    words = [w.strip().lower() for w in f if w.strip()]
word_set = set(words)

text = "Hello, thsi is a test!"
tokens = re.findall(r"[a-z']+", text.lower())  # simple word tokenizer

corrected = []
for t in tokens:
    if t in word_set:
        corrected.append(t)
    else:
        cand = difflib.get_close_matches(t, words, n=1, cutoff=0.75)
        corrected.append(cand[0] if cand else t)

print(" ".join(corrected))

Use difflib correctly: call get_close_matches once per unknown token and tune cutoff or n to control aggressiveness. For many lookups or very large dictionaries consider faster approaches: a fast C extension for edit distance (python-Levenshtein), a BK-tree for metric-space pruning, or SymSpell (symmetric-delete) which precomputes deletes for sub-millisecond lookups. These options trade more preprocessing or memory for much faster query-time performance. (docs.python.org)

Final notes: preserve original punctuation/case by mapping corrected lower-case tokens back to the original spans, cache recent corrections (LRU) for repeated typos, and avoid using built-in names like dict. As observed, fixing tokenization and the nested-loop logic removes most of the slowdown before you need more advanced structures.

Recommended Answers

All 6 Replies

Member Avatar for Member #361407

I now have this code but it is taking too long, how could i speed it up/ improve it

import difflib
dict = open("DictionaryE.txt", "r").readlines()
test = "Hello, thsi is a test!"

print dict[0:5]

correct = []
unique = []
test = test.lower()
list_words = test.split(' ')
l = []


for line in list_words:
    if line in dict:
        correct.append(line)
    else:
                for word in dict:
                        line = difflib.get_close_matches(line, dict)

print test
raw_input()

Are you using tabs for indents? They are screwed up!
I normally don't even bother with code that has obvious signs of tabs.
Do not use 'dict' as identifier, it is the name of a Python function.
Also, test your list, the punctuation marks are still attached:

test = "Hello, thsi is a test!"

test = test.lower()
list_words = test.split(' ')

# test the list
print(list_words)

"""
my output (notice that the punctuation marks are left in) -->
['hello,', 'thsi', 'is', 'a', 'test!']
"""
Member Avatar for Member #361407

oh yeah sorry, i was just using the idle and it does it automatically

got some new code, if it works ill post it but im still not sure that mine is the best method to do this, any improvements will be welcome

The way you have this coded, you are not only calling function difflib.get_close_matches(line, dict) for every word in your test string, but also for every word in the dictionary list. No wonder it is rather slow.

Member Avatar for Member #361407

yeah I wasnt too sure on how to use it, what would be the best way to use it?

Member Avatar for Member #361407

I now think that my best option is to use:

for line in list_words:
    if line in dicts:
        correct.append(line)
    elif difflib.SequenceMatcher(None, dicts, line) >= 0.8:
        line =

but im not sure what to put after the

line =

do you have any idea?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.