Add word list to pspell

[Dani]

I'm using pspell and I have a large text document of one word per line that I would like to add in addition to the US dictionary already being loaded. I don't need to be writing to this document so just loading it for the session would be fine, I guess.

It's really large so pspell_add_to_session() that adds one word at a time doesn't really make sense. I'm also really not sure what the format for .pws and .repl files are supposed to be.

[cereal]

The format for PWS files is one word per line, as you done. In Ubuntu the dictionaries used by pspell are in /usr/lib/aspell/, but you can use this command from a terminal to read the configuration and retrieve the correct path:

aspell config |grep -i data-dir

Regarding the replacement files:

The replacement word list has each replacement pair on its own line in the following format:
misspelled_word correction

For more info check these links:

• http://aspell.net/man-html/Format-of-the-Personal-and-Replacement-Dictionaries.html#Format-of-the-Personal-and-Replacement-Dictionaries
• http://aspell.net/man-html/Creating-an-Individual-Word-List.html#Creating-an-Individual-Word-List

[Dani]

OK this is what I'm doing now:

I have a file called word-list.aspell.lang.pws that looks like the following:

personal_ws-1.1 en 869229 [utf-8]
word1
word2
word2
word3
word4
...

And then my PHP code looks like this:

    $config = pspell_config_create('en', 'american');
    pspell_config_personal($config, APPPATH . 'controllers/includes/lib/word-list.aspell.lang.pws');
    pspell_config_mode($config, PSPELL_FAST|PSPELL_RUN_TOGETHER);
    $pspell = pspell_new_config($config);

No error messages, but when I try to spell check a word that is in my personal word list and not in the built-in dictionary, it doesn't recognize it.

[Dani]

I also just want to mention that pspell_config_personal() returns true on success or false on failure, and it is returning true.

[Dani]

I double-checked the file path 3 times and I CHMOD'ed the file to 777 since it says: "The personal wordlist. If the file does not exist, it will be created. The file should be writable by whoever PHP runs as (e.g. nobody)."

I also noticed, however, that it says "There is no error if the file does not exists. So if you are debugging, check wheter the file really exists."

I tried adding the following code to see if it would ammend the file, and no such luck.

    pspell_add_to_personal($pspell, "DaniWeb");
    pspell_save_wordlist($pspell);

[cereal]

Ok, change this:

personal_ws-1.1 en 869229 [utf-8]

First remove the square brackets, second this must match one of the *.cset files inside /usr/lib/aspell/, run this command to see the available encodings:

ls /usr/lib/aspell/*.cset

So, the above strings becomes:

personal_ws-1.1 en 869229 iso-8859-1

The path doesn't seem to be a problem, I've made a test with CI and works fine with your settings.

NOTE
Aspell does not seem to support utf-8:

the aspell utility will currently only function correctly with 8-bit encodings. I hope to provide utf-8 support in the future.

By consequence Pspell has the same problem. To use utf-8 you should consider the Enchant library: http://www.php.net/manual/en/intro.enchant.php

[Dani]

Still absolutely no luck :(

[cereal]

Hmm, I would do a test with file_exists() and make sure the file name and the path are correct, but as I said before I tried your configuration inside a controller and loads fine. So this probably does not help.

It may be the carriage return character? I just tried converting the pws file to Windows mode and it does not work anymore. While with Unix mode works fine. This can depend by the editor or by ftp transfer.

For example if the file is in a linux box, run:

cat -v word-list.aspell.lang.pws |head -n 20

the output should not return any ^M characters:

personal_ws-1.1 en 4 ^M
themouse^M
thecat^M
Daniweb^M
thedog^M

As said, the above will not work.

[Dani]

I guess I should have mentioned this earlier. I had read up that it can have issues with Windows return characters, so I previously ran dos2unix on the file. No ^M characters.

That being said, here's a random question for you. I see that in your sample file above, you have themouse, thecat, thedog, etc. Is that the sample data you used when testing out my config settings? Because I am using the flag PSPELL_RUN_TOGETHER which won't flag two valid words put together without a space. Therefore, it's possible that those words are being validated from the primary dictionary and your file isn't working either??

[Dani]

file_exists() returned 1 :(

[cereal]

No problem, I did a test to be sure, I used italian, spanish and french words and some random characters, just to see if there was a match, I also removed PSPELL_RUN_TOGETHER, but it works fine.

Here's the code: http://pastebin.com/JbPYfyrb

Is this running by terminal? Because this procedure showed me a problem:

php index.php CONTROLLER_NAME METHOD ARG1

Here's the error:

<div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">

<h4>A PHP Error was encountered</h4>

<p>Severity: Warning</p>
<p>Message:  pspell_save_wordlist(): pspell_save_wordlist() gave error: The file "/home/username/application/controllers/includes/lib/custom.pws" can not be opened for writing.</p>
<p>Filename: controllers/dw.php</p>
<p>Line Number: 16</p>

</div>

It refers to:

pspell_config_personal($config, APPPATH . 'controllers/includes/lib/custom.pws');

But the real path of the file is:

/var/www/test/application/controllers/includes/lib/custom.pws

not /home/.../ as in the error. In order to make it work from terminal I had to remove APPPATH. Anyway, tomorrow I will try a fresh install of CI, just to be sure this is not a reflection of one of my edits... :D

Hope someone else, in the meanwhile, helps you to find a solution.

[Dani]

I'm SOOOO friggin' frustrated :(

My actual path is /home/daniweb/httpdocs/application/...

I'm not intending to run this from the terminal, but I tried just now to see what would happen. No error messages at all. But no changes made to the file either :( It simply treated add_to_personal() just like it would add_to_session() (i.e. it worked for the script execution) and no persistent changes to the file. Like I said, no error messages either. File is chmod'ed to 777.

[pritaeas]

Maybe too obvious, but this is in the manual: "This function will not work unless you have pspell .11.2 and aspell .32.5 or later."

[Dani]

Yes, I meet those requirements. Also, the function is supposed to return true on success or false on failure, and it does return true back.

[Dani]

bump

Incredibly frustrated and still no success.

[Dani]

Whew! FINALLY!!

I was able to get it working by doing two things:

1. Manually specifying an absolute path to the .pws file
2. Cutting the file down to just a small fraction of its intended size

[cereal]

So it was also a size issue? Good to know.

[Dani]

Yes, it seemed to not like anywhere more than about 200 or so words.