fastNLTK
NLTK with a Rust engine.
Drop-in replacement. Same API, Same data, 10× faster.
NLTK is the standard Python NLP library — teaching, research, prototyping. It works great, but it's pure Python. Tokenizing a 50K-word document takes ~40 ms in NLTK. That's fine for one-offs, but in a pipeline it adds up fast.
fastNLTK wraps the same API calls in Rust. Change your import, get the same results.
No new dependency tree — the Rust engine lives in a single .pyd/.so file shipped
with the wheel.
# Before
import nltk
tokens = nltk.word_tokenize("The quick brown fox.")
# After
import fastnltk as nltk
tokens = nltk.word_tokenize("The quick brown fox.") # same call, 5–50× faster
All your NLTK data (corpora, models, pickles) still works. Nothing to re-download.
Benchmarks
366 Python drop-in compatibility tests against NLTK. 6 skipped (chat stdin). 3 expected failures.
Benchmarked on release builds against NLTK 3.10. Full results →
| Operation | NLTK | fastNLTK | Speedup |
|---|---|---|---|
| TextTiling tokenizer | 22237 ms | 32 ms | 704× |
| Maxent classifier training | 31.93 ms | 0.08 ms | 425× |
| edit_distance | 2.48 ms | 0.01 ms | 176× |
| windowdiff | 2.35 ms | 0.01 ms | 172× |
| pk (segmentation) | 2.19 ms | 0.02 ms | 90× |
| Treebank detokenizer | 6.70 ms | 0.12 ms | 55× |
| VADER sentiment | 67.06 ms | 1.75 ms | 38× |
| sentence tokenizer (Punkt) | 14.65 ms | 0.44 ms | 33× |
| S-expression tokenizer | 0.36 ms | 0.01 ms | 30× |
| Expression parser | 16.47 ms | 0.55 ms | 30× |
| Tweet tokenizer | 83.96 ms | 3.31 ms | 25× |
| CFG grammar parser | 0.05 ms | 0.002 ms | 25× |
| Lancaster stemmer | 32.81 ms | 1.41 ms | 23× |
| quadgram collocations | 101.04 ms | 4.94 ms | 21× |
| Earley parser | 6.55 ms | 0.51 ms | 13× |
| Snowball stemmer | 21.84 ms | 1.79 ms | 12× |
| word tokenizer (Treebank) | 42.18 ms | 4.27 ms | 10× |
Geometric mean across 49 benchmarks: 10.1×. Module-level breakdown:
| Module | Geo Mean | Top single |
|---|---|---|
| metrics | 146× | 176× |
| sentiment | 38× | 38× |
| sem | 30× | 30× |
| parse | 18× | 25× |
| tokenize | 18× | 704× |
| collocations | 14× | 21× |
| tree | 11× | 11× |
| translate | 9× | 9× |
| stem | 9× | 23× |
| chunk | 9× | 9× |
| classify | 9× | 425× |
| cluster | 6× | 6× |
| chat | 6× | 6× |
| tag | 4× | 9× |
| probability | 4× | 5× |
| ccg | 3× | 3× |
What's accelerated
Every module that has a Rust-backed engine:
| Module | What's in Rust |
|---|---|
tokenize |
Treebank, Toktok, Tweet, Regexp, Space, MWE, TextTiling, Punkt, SExpr, Logos DFA |
stem |
Porter, Lancaster (full 124 rules), Snowball, Regexp, WordNet, ARLSTem, Cistem, ISRI, RSLP |
tag |
PerceptronTagger, TnT (integer Viterbi), HMM, Default/Unigram/Bigram/Trigram/Regexp/Affix taggers |
classify |
NaiveBayes, Maxent (GIS), TextCat |
probability |
FreqDist, ConditionalFreqDist, MLE/Laplace/Lidstone prob dists |
lm |
MLE, Lidstone, Laplace, Kneser-Ney interpolated, Witten-Bell, StupidBackoff |
collocations |
Bigram/Trigram/Quadgram finders |
metrics |
edit_distance, jaccard, windowdiff, pk, BLEU, association, agreement, Spearman |
parse |
CFG, Earley chart parser |
tree |
Tree (bracket parse, subtrees, productions, leaves) |
chunk |
RegexpParser (NP/VP IOB extraction) |
sentiment |
VADER |
sem |
FOL expression parser, model evaluation |
inference |
Tableau prover, Resolution prover, Discourse |
cluster |
K-means |
chat |
Eliza-style chatbot |
translate |
BLEU score |
Not in Rust yet? Those calls fall through to NLTK automatically. Your code still works.
Install
pip install fastnltk
Pre-built wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), Windows (x64). Python 3.10–3.13.
Make sure you have the NLTK data you need:
python -m nltk.downloader punkt averaged_perceptron_tagger wordnet
Usage
Everything lives under fastnltk with the same names and signatures as nltk.
from fastnltk import word_tokenize, pos_tag, sent_tokenize
# Sentence segmentation (Punkt, Rust)
sents = sent_tokenize("Dr. Smith left at 5 p.m. He went home.")
# → ['Dr. Smith left at 5 p.m.', 'He went home.']
# Word tokenization (Treebank, Rust)
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
# → ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']
# POS tagging (Perceptron, Rust)
tagged = pos_tag(tokens)
# → [('The', 'DT'), ('quick', 'JJ'), ...]
Drop it in as a direct NLTK replacement:
import fastnltk as nltk
# All your existing nltk.* calls now run through Rust
nltk.word_tokenize("Hello, world!")
nltk.pos_tag(["Hello", "world"])
nltk.ne_chunk(nltk.pos_tag(["John", "lives", "in", "Boston"]))
For module-level imports:
from fastnltk.stem import PorterStemmer, LancasterStemmer
from fastnltk.tag import PerceptronTagger
from fastnltk.parse import CFG, EarleyChartParser
from fastnltk.probability import FreqDist, ConditionalFreqDist
from fastnltk.lm import MLE, KneserNeyInterpolated
from fastnltk.metrics import edit_distance, jaccard_distance
from fastnltk.collocations import BigramCollocationFinder
from fastnltk.tree import Tree
# Same API as NLTK everywhere
stemmer = LancasterStemmer()
stemmer.stem("maximum") # → 'maxim'
stemmer.stem("presumably") # → 'presum'
fd = FreqDist("hello world")
fd["l"] # → 3
fd.max() # → 'l'
tagger = PerceptronTagger()
tagger.tag(["I", "love", "NLP"]) # → [('I', 'PRP'), ('love', 'VBP'), ('NLP', 'NNP')]
tree = Tree.from_string("(S (NP I/PRP) (VP love/VBP NLP/NNP))")
tree.leaves() # → ['I/PRP', 'love/VBP', 'NLP/NNP']
tree.productions() # → ['S -> NP VP', 'NP -> I/PRP', 'VP -> love/VBP NLP/NNP']
From source
git clone https://github.com/wyattferguson/fastnltk
cd fastnltk
pip install maturin
maturin develop --release
Development
pip install -e ".[dev]"
maturin develop --release
cargo test # Rust unit tests
pytest tests/ # 375 Python tests (366 pass, 6 skip, 3 xfail)
cargo fmt --all -- --check
cargo clippy --lib
ruff check fastnltk/ tests/
See CONTRIBUTING.md for full setup and PR workflow.
Compatibility
The goal is 100% drop-in. Right now 366 of 375 tests pass (6 skipped — chat bots read stdin), with only 3 expected failures:
- Earley parse tree extraction — Rust Earley finds parses but the tree structure differs from NLTK's chart-printing format (WIP)
- ConditionalFreqDist clone semantics —
freqdist()returns a copy, so mutations don't propagate back the way NLTK's reference-sharing does (design limitation) - CCG
fromstring— NLTK 3.10'sccg.chart.fromstringis broken (upstream bug)
Every critical-path API (tokenize, tag, stem, metrics, prob, parse, chunk, sentiment, classify, collocations, tree, cluster, translate, chat) is verified byte-identical to NLTK across all tested inputs.
Platform
| Platform | Arch | Wheel |
|---|---|---|
| Linux | x86_64, aarch64 | ✅ |
| macOS | x86_64, arm64 | ✅ |
| Windows | x64 | ✅ |
License
Apache 2.0. Not affiliated with NLTK or its maintainers.
Contact + Support
Created by Wyatt Ferguson
For any questions or comments heres how you can reach me:
:octopus: Follow me on Github @wyattferguson
:mailbox_with_mail: Email me at wyattxdev@duck.com
:tropical_drink: Follow on BlueSky @wyattf
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file fastnltk-0.5.0.tar.gz.
File metadata
- Download URL: fastnltk-0.5.0.tar.gz
- Upload date:
- Size: 259.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
73413e51ebc0aa4490e9d3234fa3d76805ea28d698706e554cf299d448aa58d9
|
|
| MD5 |
52fdb9a861b96286af7cda1b9304ee24
|
|
| BLAKE2b-256 |
921900456e33161203fcdb06a3dbe2c6115c7cd97d86ca99ad6cd46ae809b07b
|
Provenance
The following attestation bundles were made for fastnltk-0.5.0.tar.gz:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.0.tar.gz -
Subject digest:
73413e51ebc0aa4490e9d3234fa3d76805ea28d698706e554cf299d448aa58d9 - Sigstore transparency entry: 2188021130
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Branch / Tag:
refs/tags/v0.5.0 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.0-cp310-abi3-win_amd64.whl.
File metadata
- Download URL: fastnltk-0.5.0-cp310-abi3-win_amd64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.10+, Windows x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b28839d1d42b17fa0aac7b2e3382369c2e1513d483862f8f8d83de20c3580929
|
|
| MD5 |
9eec5a43d98d7b76e77ef40393316542
|
|
| BLAKE2b-256 |
f8e41d716475c4b06f386201a3f23f80ab7fd069708073b1fc2ca853edba6b04
|
Provenance
The following attestation bundles were made for fastnltk-0.5.0-cp310-abi3-win_amd64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.0-cp310-abi3-win_amd64.whl -
Subject digest:
b28839d1d42b17fa0aac7b2e3382369c2e1513d483862f8f8d83de20c3580929 - Sigstore transparency entry: 2188021212
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Branch / Tag:
refs/tags/v0.5.0 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: fastnltk-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2f9a6ba5841c97bf876606abada1d4306acda3ef0b215e9c6d6d66913808239b
|
|
| MD5 |
b3f4dcd7c0204b5121ca72b0af62cf05
|
|
| BLAKE2b-256 |
5ad08fd72d0a2796f63b7314a135561b44a1753a59bdb137fe7a503a4f5f3af6
|
Provenance
The following attestation bundles were made for fastnltk-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -
Subject digest:
2f9a6ba5841c97bf876606abada1d4306acda3ef0b215e9c6d6d66913808239b - Sigstore transparency entry: 2188021280
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Branch / Tag:
refs/tags/v0.5.0 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Trigger Event:
push
-
Statement type:
File details
Details for the file fastnltk-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl.
File metadata
- Download URL: fastnltk-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.10+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c7ff613ebe9d149f68ef3da03bbf293b1775f6af52c0aad652a986bf5c33de99
|
|
| MD5 |
50c3ee74249f454296144290dc54e41e
|
|
| BLAKE2b-256 |
9423ce85e4ecd997c59f1bc8f2f41bab0aa94fa3eabd8858fd7df16268558dff
|
Provenance
The following attestation bundles were made for fastnltk-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl:
Publisher:
release.yml on wyattferguson/fastNLTK
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
fastnltk-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl -
Subject digest:
c7ff613ebe9d149f68ef3da03bbf293b1775f6af52c0aad652a986bf5c33de99 - Sigstore transparency entry: 2188021364
- Sigstore integration time:
-
Permalink:
wyattferguson/fastNLTK@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Branch / Tag:
refs/tags/v0.5.0 - Owner: https://github.com/wyattferguson
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@2c443ad06413f05bcfdba7312ab3f57a31968e08 -
Trigger Event:
push
-
Statement type:
