Pure Python Spell Checking based on Peter Norvig’s blog post on setting up a simple spell checking algorithm.
It uses a Levenshtein Distance algorithm to find permutations within an edit distance of 2 from the original word. It then compares all permutations (insertions, deletions, replacements, and transpositions) to known words in a word frequency list. Those words that are found more often in the frequency list are more likely the correct results.
pyspellchecker supports multiple languages including English, Spanish, German, French, and Portuguese. For information on how the dictionaries were created and how they can be updated and improved, please see the Dictionary Creation and Updating section of the readme!
pyspellchecker supports Python 3 and Python 2.7 but, as always, Python 3 is the preferred version!
pyspellchecker allows for the setting of the Levenshtein Distance (up to two) to check. For longer words, it is highly recommended to use a distance of 1 and not the default 2. See the quickstart to find how one can change the distance parameter.
Installation
The easiest method to install is using pip:
pip install pyspellcheckerTo install from source:
git clone https://github.com/barrust/pyspellchecker.git
cd pyspellchecker
python setup.py installQuickstart
After installation, using pyspellchecker should be fairly straight forward:
from spellchecker import SpellChecker
spell = SpellChecker()
# find those words that may be misspelled
misspelled = spell.unknown(['something', 'is', 'hapenning', 'here'])
for word in misspelled:
# Get the one `most likely` answer
print(spell.correction(word))
# Get a list of `likely` options
print(spell.candidates(word))If the Word Frequency list is not to your liking, you can add additional text to generate a more appropriate list for your use case.
from spellchecker import SpellChecker
spell = SpellChecker() # loads default word frequency list
spell.word_frequency.load_text_file('./my_free_text_doc.txt')
# if I just want to make sure some words are not flagged as misspelled
spell.word_frequency.load_words(['microsoft', 'apple', 'google'])
spell.known(['microsoft', 'google']) # will return both now!If the words that you wish to check are long, it is recommended to reduce the distance to 1. This can be accomplished either when initializing the spell check class or after the fact.
from spellchecker import SpellChecker
spell = SpellChecker(distance=1) # set at initialization
# do some work on longer words
spell.distance = 2 # set the distance parameter back to the defaultDictionary Creation and Updating
The creation of the dictionaries is, unfortunately, not an exact science. I have provided a script that, given a text file of sentences (in this case from OpenSubtitles) it will generate a word frequency list based on the words found within the text. The script then attempts to *clean up* the word frequency by, for example, removing words with invalid characters (usually from other languages), removing low count terms (misspellings?) and attempts to enforce rules as available (no more than one accent per word in Spanish). Then it removes words from a list of known words that are to be removed.
The script can be found here: scripts/build_dictionary.py`. The original word frequency list parsed from OpenSubtitles can be found in the `scripts/data/` folder along with each language’s exclude text file.
Any help in updating and maintaining the dictionaries would be greatly desired. To do this, a discussion could be started on GitHub or pull requests to update the exclude file could be added. Ideas on how to add words that are missing along with a relative frequency is something that is in the works for future versions of the dictionaries.
Additional Methods
On-line documentation is available; below contains the cliff-notes version of some of the available functions:
correction(word): Returns the most probable result for the misspelled word
candidates(word): Returns a set of possible candidates for the misspelled word
known([words]): Returns those words that are in the word frequency list
unknown([words]): Returns those words that are not in the frequency list
word_probability(word): The frequency of the given word out of all words in the frequency list
The following are less likely to be needed by the user but are available:
edit_distance_1(word): Returns a set of all strings at a Levenshtein Distance of one based on the alphabet of the selected language
edit_distance_2(word): Returns a set of all strings at a Levenshtein Distance of two based on the alphabet of the selected language
Credits
Peter Norvig blog post on setting up a simple spell checking algorithm
P Lison and J Tiedemann, 2016, OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)
1 maintainer
Metadata
1 maintainer
Release files for pyspellchecker 0.5.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pyspellchecker-0.5.6.tar.gz | 2.5 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pyspellchecker-0.5.6-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 5.0 MB
Release files / pyspellchecker-0.5.6.tar.gz
| Download URL | pyspellchecker-0.5.6.tar.gz |
|---|---|
| Size | 2.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
68af08ae5ffb3ac3d9d01ef421de9310eb3aa68ab7aa5913de846e1f3c0bb76c
|
|
BLAKE2b-256 checksum How to use checksums |
8f0a63caf73f436e1ba29eafd3688bca9b985c49314d0ca9df1d168612bc2455
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.55.0 CPython/3.9.1
|
Release files / pyspellchecker-0.5.6-py2.py3-none-any.whl
| Download URL | pyspellchecker-0.5.6-py2.py3-none-any.whl |
|---|---|
| Size | 2.5 MB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
d4392b198c67a51530147ebfd36fcec75b8fd55887a203dac8a7b8c5701694db
|
|
BLAKE2b-256 checksum How to use checksums |
6f9d5bb403decde661abc6c5467319a0729d7c238e04d8217d9fef885510ec9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.55.0 CPython/3.9.1
|
