fastNLTK

NLTK with a Rust engine.
Drop-in replacement. Same API, Same data, 12× faster.

NLTK is fantastic. The API is clean, and it does just about everything you'd want from an NLP library. The only drawback being it's pure Python, and on large inputs that starts to show. A single call is fine. A million calls in a data pipeline is a different story.

fastNLTK is NLTK with the hot path rewritten in Rust. Same API, same data, same results. Just faster — 5× to 700× depending on what you're doing. No new dependencies, no YAML files, no config. Simply change your import and watch your code fly.

# Before
import nltk
tokens = nltk.word_tokenize("The quick brown fox.")

# After
import fastnltk as nltk
tokens = nltk.word_tokenize("The quick brown fox.")  # same call, 5–50× faster

Your NLTK data (corpora, models, pickles, all of it) still works. Nothing to re-download, nothing to migrate.

Benchmarks

368 Python drop-in compatibility tests against NLTK. 6 skipped (chat stdin). 1 expected failure.

Benchmarked on release builds against NLTK 3.10. Full results →

Operation	NLTK	fastNLTK	Speedup
HMM tagger	16.06 ms	0.15 ms	104×
TextTiling tokenizer	4.69 ms	0.06 ms	77×
Treebank detokenizer	11.25 ms	0.15 ms	73×
S-expression tokenizer	1.29 ms	0.02 ms	60×
Punkt sentence tokenizer	17.65 ms	0.17 ms	106×
Tweet tokenizer	68.03 ms	1.56 ms	44×
Sentiment (VADER)	30.23 ms	0.96 ms	32×
Lancaster stemmer	54.98 ms	2.15 ms	26×
CFG grammar parser	0.11 ms	0.00 ms	28×
quadgram collocations	101.73 ms	3.01 ms	34×
edit_distance	4.55 ms	0.03 ms	165×
Trigram collocations	49.87 ms	2.43 ms	21×
Snowball stemmer	44.40 ms	2.89 ms	15×
Regexp tagger	19.59 ms	1.66 ms	12×
Tree from_string	6.46 ms	0.63 ms	10×

Geometric mean across 48 benchmarks: 12.2×. Module-level breakdown:

Module	Geo Mean	Top single
metrics	137×	165×
tag	7×	104×
sentiment	34×	34×
sem	34×	34×
parse	19×	28×
tokenize	6×	120×
collocations	14×	35×
tree	12×	12×
translate	9×	9×
stem	9×	26×
chunk	8×	8×
classify	5×	562×
cluster	5×	5×
chat	4×	4×
ccg	3×	3×
probability	3×	6×

What's accelerated

Every module that has a Rust-backed engine:

Module	What's in Rust
`tokenize`	Treebank, Toktok, Tweet, Regexp, Space, MWE, TextTiling, Punkt, SExpr, Logos DFA
`stem`	Porter, Lancaster (full 124 rules), Snowball, Regexp, WordNet, ARLSTem, Cistem, ISRI, RSLP
`tag`	PerceptronTagger, HMM (integer Viterbi), TnT, Default/Unigram/Bigram/Trigram/Regexp/Affix
`classify`	NaiveBayes, Maxent (GIS), TextCat
`corpus`	PlaintextCorpusReader, TaggedCorpusReader, CategorizedPlaintextCorpusReader
`probability`	FreqDist, ConditionalFreqDist (shared references), MLE/Laplace/Lidstone prob dists
`lm`	MLE, Lidstone, Laplace, Kneser-Ney interpolated, Witten-Bell, StupidBackoff
`collocations`	Bigram/Trigram/Quadgram finders
`metrics`	edit_distance, jaccard, windowdiff, pk, BLEU, association, agreement, Spearman
`parse`	CFG, Earley chart parser
`tree`	Tree (bracket parse, subtrees, productions, leaves)
`chunk`	RegexpParser (NP/VP IOB extraction)
`sentiment`	VADER
`sem`	FOL expression parser, model evaluation
`inference`	Tableau prover, Resolution prover, Discourse
`cluster`	K-means
`chat`	Eliza-style chatbot
`translate`	BLEU score

Not in Rust yet? Those calls fall through to NLTK automatically. Your code still works.

Install

pip install fastnltk

Pre-built wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), Windows (x64). Python 3.10–3.13.

Make sure you have the NLTK data you need:

python -m nltk.downloader punkt averaged_perceptron_tagger wordnet

Usage

Everything lives under fastnltk with the same names and signatures as nltk.

from fastnltk import word_tokenize, pos_tag, sent_tokenize

# Sentence segmentation (Punkt, Rust)
sents = sent_tokenize("Dr. Smith left at 5 p.m. He went home.")
# → ['Dr. Smith left at 5 p.m.', 'He went home.']

# Word tokenization (Treebank, Rust)
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
# → ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']

# POS tagging (Perceptron, Rust)
tagged = pos_tag(tokens)
# → [('The', 'DT'), ('quick', 'JJ'), ...]

Drop it in as a direct NLTK replacement:

import fastnltk as nltk
# All your existing nltk.* calls now run through Rust
nltk.word_tokenize("Hello, world!")
nltk.pos_tag(["Hello", "world"])
nltk.ne_chunk(nltk.pos_tag(["John", "lives", "in", "Boston"]))

For module-level imports:

from fastnltk.stem import PorterStemmer, LancasterStemmer
from fastnltk.tag import PerceptronTagger
from fastnltk.parse import CFG, EarleyChartParser
from fastnltk.probability import FreqDist, ConditionalFreqDist
from fastnltk.lm import MLE, KneserNeyInterpolated
from fastnltk.metrics import edit_distance, jaccard_distance
from fastnltk.collocations import BigramCollocationFinder
from fastnltk.tree import Tree

# Same API as NLTK everywhere
stemmer = LancasterStemmer()
stemmer.stem("maximum")          # → 'maxim'
stemmer.stem("presumably")       # → 'presum'

fd = FreqDist("hello world")
fd["l"]                           # → 3
fd.max()                          # → 'l'

tagger = PerceptronTagger()
tagger.tag(["I", "love", "NLP"])  # → [('I', 'PRP'), ('love', 'VBP'), ('NLP', 'NNP')]

tree = Tree.from_string("(S (NP I/PRP) (VP love/VBP NLP/NNP))")
tree.leaves()                     # → ['I/PRP', 'love/VBP', 'NLP/NNP']
tree.productions()                # → ['S -> NP VP', 'NP -> I/PRP', 'VP -> love/VBP NLP/NNP']

From source

git clone https://github.com/wyattferguson/fastnltk
cd fastnltk
pip install maturin
maturin develop --release

Development

pip install -e ".[dev]"
maturin develop --release

cargo test          # Rust unit tests (309 pass)
pytest tests/       # 375 Python tests (368 pass, 6 skip, 1 xfail)

cargo fmt --all -- --check
cargo clippy --lib
ruff check fastnltk/ tests/

See CONTRIBUTING.md for full setup and PR workflow.

Compatibility

The goal is 100% drop-in. Right now 368 of 375 tests pass (6 skipped — chat bots read stdin), with only 1 expected failure:

CCG fromstring — NLTK 3.10's ccg.chart.fromstring is broken (upstream bug)

Every critical-path API (tokenize, tag, stem, metrics, prob, parse, chunk, sentiment, classify, collocations, tree, cluster, translate, chat) is verified byte-identical to NLTK across all tested inputs.

Platform

Platform	Arch	Wheel
Linux	x86_64, aarch64	✅
macOS	x86_64, arm64	✅
Windows	x64	✅

License

Apache 2.0. Not affiliated with NLTK or its maintainers.

Contact + Support

Created by Wyatt Ferguson

For any questions or comments heres how you can reach me:

:octopus: Follow me on Github @wyattferguson

:mailbox_with_mail: Email me at wyattxdev@duck.com

:tropical_drink: Follow on BlueSky @wyattf

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastnltk-0.5.4.tar.gz (269.2 kB view details)

Uploaded Jul 18, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

fastnltk-0.5.4-cp310-abi3-win_amd64.whl (1.4 MB view details)

Uploaded Jul 18, 2026 CPython 3.10+Windows x86-64

fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Jul 18, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Jul 18, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

File details

Details for the file fastnltk-0.5.4.tar.gz.

File metadata

Download URL: fastnltk-0.5.4.tar.gz
Upload date: Jul 18, 2026
Size: 269.2 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.5.4.tar.gz
Algorithm	Hash digest
SHA256	`8d338f7d4071de94562c1c109a25062c3d1eef4df6d06c34d9715a075d3658b6`
MD5	`5002bf7b0a6e5153a0373c67bbb4d0c3`
BLAKE2b-256	`813ecabd0ede64dab343ca022bd5098536c990f2add6a66c158483024976f646`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.5.4.tar.gz:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.5.4.tar.gz
- Subject digest: 8d338f7d4071de94562c1c109a25062c3d1eef4df6d06c34d9715a075d3658b6
- Sigstore transparency entry: 2195376659
- Sigstore integration time: Jul 18, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Branch / Tag: refs/tags/v0.5.4
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Trigger Event: push

File details

Details for the file fastnltk-0.5.4-cp310-abi3-win_amd64.whl.

File metadata

Download URL: fastnltk-0.5.4-cp310-abi3-win_amd64.whl
Upload date: Jul 18, 2026
Size: 1.4 MB
Tags: CPython 3.10+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.5.4-cp310-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`647ae9bc62e18f20186e53ad973f1ab23cd6aa1d64f6816ba15cdd4f3338c2f7`
MD5	`656b0e04c25e341c8312f528fb175acd`
BLAKE2b-256	`1c2c178d82082cd238cddf3ec4eca3cc3f00570c7952598ce46d98f6e1fa7a4b`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.5.4-cp310-abi3-win_amd64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.5.4-cp310-abi3-win_amd64.whl
- Subject digest: 647ae9bc62e18f20186e53ad973f1ab23cd6aa1d64f6816ba15cdd4f3338c2f7
- Sigstore transparency entry: 2195376660
- Sigstore integration time: Jul 18, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Branch / Tag: refs/tags/v0.5.4
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Trigger Event: push

File details

Details for the file fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Jul 18, 2026
Size: 1.6 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`0dc25edb50e7af41e428d8d65bcc7115452e9f63f661582f98a8265b6e715bc3`
MD5	`8b6c2b645199bdb7d8573470efc2f261`
BLAKE2b-256	`bbd7948a53a2e321694ca62f01052c2f3b7f8402df1208dc14ca4f58347af9fe`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.5.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 0dc25edb50e7af41e428d8d65bcc7115452e9f63f661582f98a8265b6e715bc3
- Sigstore transparency entry: 2195376665
- Sigstore integration time: Jul 18, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Branch / Tag: refs/tags/v0.5.4
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Trigger Event: push

File details

Details for the file fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Jul 18, 2026
Size: 1.5 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`7919d05222ad341775bb416d4a54695546dca4e1d554d3a503e502cc6dc4ae99`
MD5	`4630a7d99c23ba3cc2e197ecbe9320bf`
BLAKE2b-256	`9fc4c20aeb4abe697de526ed6cc32de1fc344464ae5c9170407c3f809be7f362`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.5.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: 7919d05222ad341775bb416d4a54695546dca4e1d554d3a503e502cc6dc4ae99
- Sigstore transparency entry: 2195376663
- Sigstore integration time: Jul 18, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Branch / Tag: refs/tags/v0.5.4
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@f9895fa033370b23ee6b7d6edde2ac86c1fb8256
- Trigger Event: push

Release history Release notifications | RSS feed

0.5.5

Aug 5, 2026 4 files

This release

0.5.4 This release

Jul 18, 2026 4 files

Jul 18, 2026 4 files

Jul 17, 2026 4 files

Jul 17, 2026 4 files

Jul 16, 2026 6 files

Jul 15, 2026 4 files

fastnltk 0.5.4

fastNLTK

Benchmarks

What's accelerated

Install

Usage

From source

Development

Compatibility

Platform

License

Contact + Support

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed