fastNLTK
NLTK with a Rust engine.
Drop-in replacement. Same API, Same data, 12× faster.
NLTK is fantastic. The API is clean, and it does just about everything you'd want from an NLP library. The only drawback being it's pure Python, and on large inputs that starts to show. A single call is fine. A million calls in a data pipeline is a different story.
fastNLTK is NLTK with the hot path rewritten in Rust. Same API, same data, same results. Just faster — 5× to 700× depending on what you're doing. No new dependencies, no YAML files, no config. Simply change your import and watch your code fly.
# Before
import nltk
tokens = nltk.word_tokenize("The quick brown fox.")
# After
import fastnltk as nltk
tokens = nltk.word_tokenize("The quick brown fox.") # same call, 5–50× faster
Your NLTK data (corpora, models, pickles, all of it) still works. Nothing to re-download, nothing to migrate.
Benchmarks
368 Python drop-in compatibility tests against NLTK. 6 skipped (chat stdin). 1 expected failure.
Benchmarked on release builds against NLTK 3.10. Full results →
| Operation | NLTK | fastNLTK | Speedup |
|---|---|---|---|
| HMM tagger | 16.06 ms | 0.15 ms | 104× |
| TextTiling tokenizer | 4.69 ms | 0.06 ms | 77× |
| Treebank detokenizer | 11.25 ms | 0.15 ms | 73× |
| S-expression tokenizer | 1.29 ms | 0.02 ms | 60× |
| Punkt sentence tokenizer | 17.65 ms | 0.17 ms | 106× |
| Tweet tokenizer | 68.03 ms | 1.56 ms | 44× |
| Sentiment (VADER) | 30.23 ms | 0.96 ms | 32× |
| Lancaster stemmer | 54.98 ms | 2.15 ms | 26× |
| CFG grammar parser | 0.11 ms | 0.00 ms | 28× |
| quadgram collocations | 101.73 ms | 3.01 ms | 34× |
| edit_distance | 4.55 ms | 0.03 ms | 165× |
| Trigram collocations | 49.87 ms | 2.43 ms | 21× |
| Snowball stemmer | 44.40 ms | 2.89 ms | 15× |
| Regexp tagger | 19.59 ms | 1.66 ms | 12× |
| Tree from_string | 6.46 ms | 0.63 ms | 10× |
Geometric mean across 48 benchmarks: 12.2×. Module-level breakdown:
| Module | Geo Mean | Top single |
|---|---|---|
| metrics | 137× | 165× |
| tag | 7× | 104× |
| sentiment | 34× | 34× |
| sem | 34× | 34× |
| parse | 19× | 28× |
| tokenize | 6× | 120× |
| collocations | 14× | 35× |
| tree | 12× | 12× |
| translate | 9× | 9× |
| stem | 9× | 26× |
| chunk | 8× | 8× |
| classify | 5× | 562× |
| cluster | 5× | 5× |
| chat | 4× | 4× |
| ccg | 3× | 3× |
| probability | 3× | 6× |
What's accelerated
Every module that has a Rust-backed engine:
| Module | What's in Rust |
|---|---|
tokenize |
Treebank, Toktok, Tweet, Regexp, Space, MWE, TextTiling, Punkt, SExpr, Logos DFA |
stem |
Porter, Lancaster (full 124 rules), Snowball, Regexp, WordNet, ARLSTem, Cistem, ISRI, RSLP |
tag |
PerceptronTagger, HMM (integer Viterbi), TnT, Default/Unigram/Bigram/Trigram/Regexp/Affix |
classify |
NaiveBayes, Maxent (GIS), TextCat |
corpus |
PlaintextCorpusReader, TaggedCorpusReader, CategorizedPlaintextCorpusReader |
probability |
FreqDist, ConditionalFreqDist (shared references), MLE/Laplace/Lidstone prob dists |
lm |
MLE, Lidstone, Laplace, Kneser-Ney interpolated, Witten-Bell, StupidBackoff |
collocations |
Bigram/Trigram/Quadgram finders |
metrics |
edit_distance, jaccard, windowdiff, pk, BLEU, association, agreement, Spearman |
parse |
CFG, Earley chart parser |
tree |
Tree (bracket parse, subtrees, productions, leaves) |
chunk |
RegexpParser (NP/VP IOB extraction) |
sentiment |
VADER |
sem |
FOL expression parser, model evaluation |
inference |
Tableau prover, Resolution prover, Discourse |
cluster |
K-means |
chat |
Eliza-style chatbot |
translate |
BLEU score |
Not in Rust yet? Those calls fall through to NLTK automatically. Your code still works.
Install
pip install fastnltk
Pre-built wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), Windows (x64). Python 3.10–3.13.
Make sure you have the NLTK data you need:
python -m nltk.downloader punkt averaged_perceptron_tagger wordnet
Usage
Everything lives under fastnltk with the same names and signatures as nltk.
from fastnltk import word_tokenize, pos_tag, sent_tokenize
# Sentence segmentation (Punkt, Rust)
sents = sent_tokenize("Dr. Smith left at 5 p.m. He went home.")
# → ['Dr. Smith left at 5 p.m.', 'He went home.']
# Word tokenization (Treebank, Rust)
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
# → ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']
# POS tagging (Perceptron, Rust)
tagged = pos_tag(tokens)
# → [('The', 'DT'), ('quick', 'JJ'), ...]
Drop it in as a direct NLTK replacement:
import fastnltk as nltk
# All your existing nltk.* calls now run through Rust
nltk.word_tokenize("Hello, world!")
nltk.pos_tag(["Hello", "world"])
nltk.ne_chunk(nltk.pos_tag(["John", "lives", "in", "Boston"]))
For module-level imports:
from fastnltk.stem import PorterStemmer, LancasterStemmer
from fastnltk.tag import PerceptronTagger
from fastnltk.parse import CFG, EarleyChartParser
from fastnltk.probability import FreqDist, ConditionalFreqDist
from fastnltk.lm import MLE, KneserNeyInterpolated
from fastnltk.metrics import edit_distance, jaccard_distance
from fastnltk.collocations import BigramCollocationFinder
from fastnltk.tree import Tree
# Same API as NLTK everywhere
stemmer = LancasterStemmer()
stemmer.stem("maximum") # → 'maxim'
stemmer.stem("presumably") # → 'presum'
fd = FreqDist("hello world")
fd["l"] # → 3
fd.max() # → 'l'
tagger = PerceptronTagger()
tagger.tag(["I", "love", "NLP"]) # → [('I', 'PRP'), ('love', 'VBP'), ('NLP', 'NNP')]
tree = Tree.from_string("(S (NP I/PRP) (VP love/VBP NLP/NNP))")
tree.leaves() # → ['I/PRP', 'love/VBP', 'NLP/NNP']
tree.productions() # → ['S -> NP VP', 'NP -> I/PRP', 'VP -> love/VBP NLP/NNP']
From source
git clone https://github.com/wyattferguson/fastnltk
cd fastnltk
pip install maturin
maturin develop --release
Development
pip install -e ".[dev]"
maturin develop --release
cargo test # Rust unit tests (309 pass)
pytest tests/ # 375 Python tests (368 pass, 6 skip, 1 xfail)
cargo fmt --all -- --check
cargo clippy --lib
ruff check fastnltk/ tests/
See CONTRIBUTING.md for full setup and PR workflow.
Compatibility
The goal is 100% drop-in. Right now 368 of 375 tests pass (6 skipped — chat bots read stdin), with only 1 expected failure:
- CCG
fromstring— NLTK 3.10'sccg.chart.fromstringis broken (upstream bug)
Every critical-path API (tokenize, tag, stem, metrics, prob, parse, chunk, sentiment, classify, collocations, tree, cluster, translate, chat) is verified byte-identical to NLTK across all tested inputs.
Platform
| Platform | Arch | Wheel |
|---|---|---|
| Linux | x86_64, aarch64 | ✅ |
| macOS | x86_64, arm64 | ✅ |
| Windows | x64 | ✅ |
License
Apache 2.0. Not affiliated with NLTK or its maintainers.
Contact + Support
Created by Wyatt Ferguson
For any questions or comments heres how you can reach me:
:octopus: Follow me on Github @wyattferguson
:mailbox_with_mail: Email me at wyattxdev@duck.com
:tropical_drink: Follow on BlueSky @wyattf
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file fastnltk-0.5.3.tar.gz.
File metadata
- Download URL: fastnltk-0.5.3.tar.gz
- Upload date:
- Size: 268.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c9cb7116082ae0e8bf25d90587fae79e22a62d66b3ff0890ad95b4779d9932eb
|
|
| MD5 |
a92e1d746ae0bc9ad2ec93ab2c0b2a7a
|
|
| BLAKE2b-256 |
bb46c716ea36761a6b59de82381b4a809235e36626b1baab09f9425f6065f0f5
|
Provenance
The following attestation bundles were made for fastnltk-0.5.3.tar.gz:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.3.tar.gz -
Subject digest:
c9cb7116082ae0e8bf25d90587fae79e22a62d66b3ff0890ad95b4779d9932eb - Sigstore transparency entry: 2193186112
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Branch / Tag:
refs/tags/v0.5.3 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.3-cp310-abi3-win_amd64.whl.
File metadata
- Download URL: fastnltk-0.5.3-cp310-abi3-win_amd64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.10+, Windows x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
61ed4c4edc4fc795c1d102f4514b37c9a6d88971d768c079d01e86c6fb0d7304
|
|
| MD5 |
a5d3ab9f9b7a70bec8701c4fc715eadb
|
|
| BLAKE2b-256 |
abf24138f2a4dc1fe74519227198ce8e053aaaccf909a3af91a4e197ef777b6f
|
Provenance
The following attestation bundles were made for fastnltk-0.5.3-cp310-abi3-win_amd64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.3-cp310-abi3-win_amd64.whl -
Subject digest:
61ed4c4edc4fc795c1d102f4514b37c9a6d88971d768c079d01e86c6fb0d7304 - Sigstore transparency entry: 2193186117
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Branch / Tag:
refs/tags/v0.5.3 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: fastnltk-0.5.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
cb92e24d376ee0c9102919dfa1f1e6a91517ac5d9e3d8d44625323a820d6842d
|
|
| MD5 |
879d7f94018354eacec4a3be5e38aabb
|
|
| BLAKE2b-256 |
580a22266da1062d3f065cff7d26df19e1357dfe4d340bb9b7aaa472cba42c71
|
Provenance
The following attestation bundles were made for fastnltk-0.5.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -
Subject digest:
cb92e24d376ee0c9102919dfa1f1e6a91517ac5d9e3d8d44625323a820d6842d - Sigstore transparency entry: 2193186133
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Branch / Tag:
refs/tags/v0.5.3 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: fastnltk-0.5.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3b632dee378f9877d9f3d81370cdd77af219068c7430e9d0c17c20cc3ff8f541
|
|
| MD5 |
0b2cb13d48461b0bca840f39f1ae4637
|
|
| BLAKE2b-256 |
0a9e0a88f5a9bd8e389f79e86a896db7f498f5949c8a0eb5e6154582589289ba
|
Provenance
The following attestation bundles were made for fastnltk-0.5.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -
Subject digest:
3b632dee378f9877d9f3d81370cdd77af219068c7430e9d0c17c20cc3ff8f541 - Sigstore transparency entry: 2193186126
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Branch / Tag:
refs/tags/v0.5.3 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@fdc1260bc70303a035a22f0379ec9dc7dc6eb791 -
Trigger Event:
push
-
Statement type:
