Member Avatar for VernonDozier

I have developed a program for a word game that reads from a text file containing a list of all legal English words, then stores those words in an array used by other parts of the program. The program works, but I am not very happy with the word list that I have used as a text file since it seems to contains words that don't exist like "abd", acronyms, several-word phrases, proper names, chemistry elements like "Cm", and worst of all, swear words. I have found many a word list, but they all seem to either be specialized word lists, contain only commonly used words, contain arcane words that haven't been used since the 15th century, contain words that appear to not exist, contain acronyms or proper names, or most bizarrely, truncate words after eight letters.

I am looking for a regular old text file containing what would be found in a large modern dictionary like Merriam Webster, which has a good online dictionary but (not surprisingly) doesn't furnish a text file of their dictionary. I don't need the definitions, only the words themselves. Any ideas? Thanks.

  • 9 Contributors
  • 12 Replies
  • 4K Views
  • 5 Years Discussion Span
  • Latest Post Latest Post by Saumil11

Dani AI

Generated

VernonDozier, you are running into the classic problem that there is no single, perfect, redistribution-friendly list of “all legal English words,” and dialect and editorial policy matter (Ancient Dragon was right). A practical approach is to start from a permissively licensed base list designed for spellcheckers, then filter it to your game’s rules. SCOWL (Spell Checker Oriented Word Lists) is widely used for this: you can pick American/British/Canadian/Australian variants, control how inclusive the list is by frequency “levels,” and avoid categories you do not want (proper names, abbreviations, etc.). It is available as plain wordlists under permissive terms and is intended for exactly this kind of customization. ()

If you want a second signal to weed out oddities like “abd,” intersect SCOWL with WordNet lemmas. WordNet is freely redistributable with attribution; keeping only words that appear as lemmas can remove many abbreviations and nonce forms while still retaining inflections you care about. (wordnet.princeton.edu)

Legal note on Scrabble lists (replies above mention SOWPODS/TWL): the current North American tournament list (NWL2023) is licensed by NASPA. Redistribution in apps or as text generally requires permission and, for commercial use, a license. If you need a Scrabble-compatible list, contact NASPA; otherwise, avoid scraping or republishing those files. (www2.scrabbleplayers.org)

Minimal C++ filter pipeline (post-process SCOWL):

#include <bits/stdc++.h>
using namespace std;
int main() {
    ios::sync_with_stdio(false);
    cin.tie(nullptr);
    regex ok("^[a-z]+$");            // letters only, lowercase
    unordered_set<string> dict;
    string w;
    while (cin >> w) {
        transform(w.begin(), w.end(), w.begin(), ::tolower);
        if (regex_match(w, ok) && w.size() >= 2 && w.size() <= 15)
            dict.insert(w);
    }
    for (auto &s : dict) cout << s << '\n';
}

From there, optionally intersect with a WordNet lemma list and/or drop words below a frequency threshold if you want a “modern usage” feel. (wordnet.princeton.edu)

Recommended Answers

All 12 Replies

Member Avatar for vijayan121

wouldn't /usr/share/dict/words suffice? in FreeBSD, this has words from Webster's 2nd International dictionary (quite a few words).

>cat /usr/share/dict/words | wc -lw
  235882  235882

if you are not on unix, download from http://www.freebsd.org/cgi/cvsweb.cgi/src/share/dict/web2

Member Avatar for VernonDozier

Thanks for the tip, vijayan121. It's better than the one I had for sure. It still has a bunch of "words" with numbers in them, proper names, acronyms, and words like "abd", which I'm pretty sure is not a word. However, like I said, it's better than what I have now and if need be, I can write a little program that strips out words with digits, capital letters, etc., and get a cleaner version. That'll get most of the dubious words out, but I am worried about the words like "abd". Before I do that, though, does anyone else have any leads on a cleaner word list?

Thanks. Definitely better off than I was before.

Member Avatar for Ancient Dragon

I doubt you will ever find a perfect word list -- afterall even Websters dictionary contains foreign words, acronyms, abbreviations etc. Then there's the issue of whose version of English do you want to use? Oxford English, The Queen's English, American English, etc. Then what will you do with all the foreign words ?

Member Avatar for VernonDozier

O.K. I guess since it's my game I get to make the call as to what is a word and what is not. With my vocabulary, I'm not sure I'm up to the job. ;) I think I'll just weed out the ones with capital letters and digits, phrases, periods, etc., and maybe put the resulting document through spell-check and hopefully there'll be few enough "misspellings" to take it on a case by case basis. Thanks guys.

Member Avatar for vijayan121

you may want to have a look at and if you are interested in american (gotta and the like),

Member Avatar for sudon't

SOWPODS, the Scrabble dictionary, might be just the thing. If it's 'legal' in Scrabble, should be legal for your game as well. Just search for "SOWPODS".

Member Avatar for mike_2000_17

I doubt that Vernon is still developing this game, four years later. But the scrabble dictionary is a good suggestion that others might benefit from. For that reason, I won't vote down your post, but be careful in the future not to resurrect threads from years past.

Member Avatar for Member #46692

This thread is old!

Member Avatar for cbielich

It can stil help someone :)

Member Avatar for PokeHerOne

This has helped me :) Cheers all

Member Avatar for Saumil11

thanks bub u made my day.......

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.