NLTK is fantastic. The API is clean, and it does just about everything you'd want from an NLP library. The only drawback being it's pure Python, and on large inputs that starts to show. A single call is fine. A million calls in a data pipeline is a different story.

fastNLTK is NLTK with the hot path rewritten in Rust. Same API, same data, same results. Just faster — 5× to 455× depending on what you're doing. No new dependencies, no YAML files, no config. Simply change your import and watch your code fly.

# Before
import nltk
tokens = nltk.word_tokenize("The quick brown fox.")

# After
import fastnltk as nltk
tokens = nltk.word_tokenize("The quick brown fox.")  # same call, 5–50× faster

Your NLTK data (corpora, models, pickles, all of it) still works. Nothing to re-download, nothing to migrate.

Benchmarks

368 Python drop-in compatibility tests against NLTK. 6 skipped (chat stdin). 1 expected failure.

Benchmarked on release builds against NLTK 3.10. Full results →

Geometric mean across 51 benchmarks: 9.5×. Module-level breakdown:

What's accelerated

Every module that has a Rust-backed engine:

Not in Rust yet? Those calls fall through to NLTK automatically. Your code still works.

Install

pip install fastnltk

Pre-built wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), Windows (x64). Python 3.10–3.13.

Make sure you have the NLTK data you need:

python -m nltk.downloader punkt averaged_perceptron_tagger wordnet

Usage

Everything lives under fastnltk with the same names and signatures as nltk.

from fastnltk import word_tokenize, pos_tag, sent_tokenize

# Sentence segmentation (Punkt, Rust)
sents = sent_tokenize("Dr. Smith left at 5 p.m. He went home.")
# → ['Dr. Smith left at 5 p.m.', 'He went home.']

# Word tokenization (Treebank, Rust)
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
# → ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']

# POS tagging (Perceptron, Rust)
tagged = pos_tag(tokens)
# → [('The', 'DT'), ('quick', 'JJ'), ...]

Drop it in as a direct NLTK replacement:

import fastnltk as nltk
# All your existing nltk.* calls now run through Rust
nltk.word_tokenize("Hello, world!")
nltk.pos_tag(["Hello", "world"])
nltk.ne_chunk(nltk.pos_tag(["John", "lives", "in", "Boston"]))

For module-level imports:

from fastnltk.stem import PorterStemmer, LancasterStemmer
from fastnltk.tag import PerceptronTagger
from fastnltk.parse import CFG, EarleyChartParser
from fastnltk.probability import FreqDist, ConditionalFreqDist
from fastnltk.lm import MLE, KneserNeyInterpolated
from fastnltk.metrics import edit_distance, jaccard_distance
from fastnltk.collocations import BigramCollocationFinder
from fastnltk.tree import Tree

# Same API as NLTK everywhere
stemmer = LancasterStemmer()
stemmer.stem("maximum")          # → 'maxim'
stemmer.stem("presumably")       # → 'presum'

fd = FreqDist("hello world")
fd["l"]                           # → 3
fd.max()                          # → 'l'

tagger = PerceptronTagger()
tagger.tag(["I", "love", "NLP"])  # → [('I', 'PRP'), ('love', 'VBP'), ('NLP', 'NNP')]

tree = Tree.from_string("(S (NP I/PRP) (VP love/VBP NLP/NNP))")
tree.leaves()                     # → ['I/PRP', 'love/VBP', 'NLP/NNP']
tree.productions()                # → ['S -> NP VP', 'NP -> I/PRP', 'VP -> love/VBP NLP/NNP']

From source

git clone https://github.com/wyattferguson/fastnltk
cd fastnltk
pip install maturin
maturin develop --release

Development

pip install -e ".[dev]"
maturin develop --release

cargo test          # Rust unit tests (309 pass)
pytest tests/       # 375 Python tests (368 pass, 6 skip, 1 xfail)

cargo fmt --all -- --check
cargo clippy --lib
ruff check fastnltk/ tests/

See CONTRIBUTING.md for full setup and PR workflow.

Compatibility

The goal is 100% drop-in. Right now 368 of 375 tests pass (6 skipped — chat bots read stdin), with only 1 expected failure:

• CCG fromstring — NLTK 3.10's ccg.chart.fromstring is broken (upstream bug)

Every critical-path API (tokenize, tag, stem, metrics, prob, parse, chunk, sentiment, classify, collocations, tree, cluster, translate, chat) is verified byte-identical to NLTK across all tested inputs.

Platform

License

Apache 2.0. Not affiliated with NLTK or its maintainers.

Contact + Support

Created by Wyatt Ferguson

For any questions or comments heres how you can reach me:

🐙 Follow me on Github @wyattferguson

📬 Email me at wyattxdev@duck.com

🍹 Follow on BlueSky @wyattf