Snowball stemming library collection for Python
Python 3 (>= 3.3) is supported. We no longer actively support Python 2 as the Python developers stopped supporting it at the start of 2020. Snowball 2.1.0 was the last release to officially support Python 2.
What is Stemming?
Stemming maps different forms of the same word to a common “stem” - for example, the English stemmer maps connection, connections, connective, connected, and connecting to connect. So a searching for connected would also find documents which only have the other forms.
This stem form is often a word itself, but this is not always the case as this is not a requirement for text search systems, which are the intended field of use. We also aim to conflate words with the same meaning, rather than all words with a common linguistic root (so awe and awful don’t have the same stem), and over-stemming is more problematic than under-stemming so we tend not to stem in cases that are hard to resolve. If you want to always reduce words to a root form and/or get a root form which is itself a word then Snowball’s stemming algorithms likely aren’t the right answer.
How to use library
The snowballstemmer module has two functions.
The snowballstemmer.algorithms function returns a list of available algorithm names.
The snowballstemmer.stemmer function takes an algorithm name and returns a Stemmer object.
Stemmer objects have a Stemmer.stemWord(word) method and a Stemmer.stemWords(word[]) method.
import snowballstemmer
stemmer = snowballstemmer.stemmer('english');
print(stemmer.stemWords("We are the world".split()));Automatic Acceleration
PyStemmer is a wrapper module for Snowball’s libstemmer_c and should provide results 100% compatible to snowballstemmer.
PyStemmer is faster because it wraps generated C versions of the stemmers; snowballstemmer uses generate Python code and is slower but offers a pure Python solution.
If PyStemmer is installed, snowballstemmer.stemmer returns a PyStemmer Stemmer object which provides the same Stemmer.stemWord() and Stemmer.stemWords() methods.
Benchmark
This is a crude benchmark which measures the time for running each stemmer on every word in its sample vocabulary (10,787,583 words over 26 languages). It’s not a realistic test of normal use as a real application would do much more than just stemming. It’s also skewed towards the stemmers which do more work per word and towards those with larger sample vocabularies.
Python 2.7 + snowballstemmer : 13m00s (15.0 * PyStemmer)
Python 3.7 + snowballstemmer : 12m19s (14.2 * PyStemmer)
PyPy 7.1.1 (Python 2.7.13) + snowballstemmer : 2m14s (2.6 * PyStemmer)
PyPy 7.1.1 (Python 3.6.1) + snowballstemmer : 1m46s (2.0 * PyStemmer)
Python 2.7 + PyStemmer : 52s
For reference the equivalent test for C runs in 9 seconds.
These results are for Snowball 2.0.0. They’re likely to evolve over time as the code Snowball generates for both Python and C continues to improve (for a much older test over a different set of stemmers using Python 2.7, snowballstemmer was 30 times slower than PyStemmer, or 9 times slower with PyPy).
The message to take away is that if you’re stemming a lot of words you should either install PyStemmer (which snowballstemmer will then automatically use for you as described above) or use PyPy.
The TestApp example
The testapp.py example program allows you to run any of the stemmers on a sample vocabulary.
Usage:
testapp.py <algorithm> "sentences ... "
$ python testapp.py English "sentences... "
Release files for snowballstemmer 2.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| snowballstemmer-2.2.0.tar.gz | 86.7 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| snowballstemmer-2.2.0-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 179.7 kB
Release files / snowballstemmer-2.2.0.tar.gz
| Download URL | snowballstemmer-2.2.0.tar.gz |
|---|---|
| Size | 86.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
09b16deb8547d3412ad7b590689584cd0fe25ec8db3be37788be3810cbf19cb1
|
|
BLAKE2b-256 checksum How to use checksums |
447baf302bebf22c749c56c9c3e8ae13190b5b5db37a33d9068652e8f73b7089
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0+
|
Release files / snowballstemmer-2.2.0-py2.py3-none-any.whl
| Download URL | snowballstemmer-2.2.0-py2.py3-none-any.whl |
|---|---|
| Size | 93.0 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
c8e1716e83cc398ae16824e5572ae04e0d9fc2c6b985fb0f900f5f0c96ecba1a
|
|
BLAKE2b-256 checksum How to use checksums |
eddcc02e01294f7265e63a7315fe086dd1df7dacb9f840a804da846b96d01b96
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0+
|
Security for snowballstemmer 2.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
