fastNLTK

NLTK with a Rust engine.
Drop-in replacement. Same API, Same data, 10× faster.

NLTK is the standard Python NLP library — teaching, research, prototyping. It works great, but it's pure Python. Tokenizing a 50K-word document takes ~40 ms in NLTK. That's fine for one-offs, but in a pipeline it adds up fast.

fastNLTK wraps the same API calls in Rust. Change your import, get the same results. No new dependency tree — the Rust engine lives in a single .pyd/.so file shipped with the wheel.

# Before
import nltk
tokens = nltk.word_tokenize("The quick brown fox.")

# After
import fastnltk as nltk
tokens = nltk.word_tokenize("The quick brown fox.")  # same call, 5–50× faster

All your NLTK data (corpora, models, pickles) still works. Nothing to re-download.

Benchmarks

309 Rust unit tests. 331 drop-in compatibility tests against NLTK. 0 failures.

Benchmarked on release builds against NLTK 3.10, Rust 1.97.1. Full results →

Operation	NLTK	fastNLTK	Speedup
TextTiling tokenizer	35000 ms	48 ms	732×
edit_distance	3.40 ms	0.01 ms	255×
windowdiff	2.94 ms	0.01 ms	211×
pk (segmentation)	2.79 ms	0.03 ms	109×
Maxent classifier training	69.00 ms	0.15 ms	464×
sentence tokenizer (Punkt)	35.49 ms	0.59 ms	60×
Treebank detokenizer	9.07 ms	0.19 ms	48×
VADER sentiment	116.83 ms	2.52 ms	46×
S-expression tokenizer	0.55 ms	0.01 ms	46×
CFG grammar parser	0.11 ms	0.002 ms	43×
Expression parser	36.90 ms	0.89 ms	42×
Tweet tokenizer	137.89 ms	4.71 ms	29×
quadgram collocations	168.95 ms	6.48 ms	26×
Lancaster stemmer	56.34 ms	2.59 ms	22×
Earley parser	17.01 ms	0.87 ms	20×
Snowball stemmer	39.20 ms	2.81 ms	14×
word tokenizer (Treebank)	55.27 ms	5.87 ms	9×

Geometric mean across 49 benchmarks: 11.2×. Module-level breakdown:

Module	Geo Mean	Top single
metrics	170×	255×
sentiment	46×	46×
sem	42×	42×
parse	29×	43×
tokenize	18×	732×
collocations	17×	26×
translate	16×	16×
tree	13×	13×
chunk	9×	9×
stem	9×	22×
classify	9×	464×
cluster	9×	9×
tag	4×	10×
probability	4×	5×
ccg	2×	2×

What's accelerated

Every module that has a Rust-backed engine:

Module	What's in Rust
`tokenize`	Treebank, Toktok, Tweet, Regexp, Space, MWE, TextTiling, Punkt, SExpr, Logos DFA
`stem`	Porter, Lancaster (full 124 rules), Snowball, Regexp, WordNet, ARLSTem, Cistem, ISRI, RSLP
`tag`	PerceptronTagger, TnT (integer Viterbi), HMM, Default/Unigram/Bigram/Trigram/Regexp/Affix taggers
`classify`	NaiveBayes, Maxent (GIS), TextCat
`probability`	FreqDist, ConditionalFreqDist, MLE/Laplace/Lidstone prob dists
`lm`	MLE, Lidstone, Laplace, Kneser-Ney interpolated, Witten-Bell, StupidBackoff
`collocations`	Bigram/Trigram/Quadgram finders
`metrics`	edit_distance, jaccard, windowdiff, pk, BLEU, association, agreement, Spearman
`parse`	CFG, Earley chart parser
`tree`	Tree (bracket parse, subtrees, productions, leaves)
`chunk`	RegexpParser (NP/VP IOB extraction)
`sentiment`	VADER
`sem`	FOL expression parser, model evaluation
`inference`	Tableau prover, Resolution prover, Discourse
`cluster`	K-means
`chat`	Eliza-style chatbot
`translate`	BLEU score

Not in Rust yet? Those calls fall through to NLTK automatically. Your code still works.

Install

pip install fastnltk

Pre-built wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), Windows (x64). Python 3.10–3.13.

Make sure you have the NLTK data you need:

python -m nltk.downloader punkt averaged_perceptron_tagger wordnet

Usage

Everything lives under fastnltk with the same names and signatures as nltk.

from fastnltk import word_tokenize, pos_tag, sent_tokenize

# Sentence segmentation (Punkt, Rust)
sents = sent_tokenize("Dr. Smith left at 5 p.m. He went home.")
# → ['Dr. Smith left at 5 p.m.', 'He went home.']

# Word tokenization (Treebank, Rust)
tokens = word_tokenize("The quick brown fox jumps over the lazy dog.")
# → ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.']

# POS tagging (Perceptron, Rust)
tagged = pos_tag(tokens)
# → [('The', 'DT'), ('quick', 'JJ'), ...]

Drop it in as a direct NLTK replacement:

import fastnltk as nltk
# All your existing nltk.* calls now run through Rust
nltk.word_tokenize("Hello, world!")
nltk.pos_tag(["Hello", "world"])
nltk.ne_chunk(nltk.pos_tag(["John", "lives", "in", "Boston"]))

For module-level imports:

from fastnltk.stem import PorterStemmer, LancasterStemmer
from fastnltk.tag import PerceptronTagger
from fastnltk.parse import CFG, EarleyChartParser
from fastnltk.probability import FreqDist, ConditionalFreqDist
from fastnltk.lm import MLE, KneserNeyInterpolated
from fastnltk.metrics import edit_distance, jaccard_distance
from fastnltk.collocations import BigramCollocationFinder
from fastnltk.tree import Tree

# Same API as NLTK everywhere
stemmer = LancasterStemmer()
stemmer.stem("maximum")          # → 'maxim'
stemmer.stem("presumably")       # → 'presum'

fd = FreqDist("hello world")
fd["l"]                           # → 3
fd.max()                          # → 'l'

tagger = PerceptronTagger()
tagger.tag(["I", "love", "NLP"])  # → [('I', 'PRP'), ('love', 'VBP'), ('NLP', 'NNP')]

tree = Tree.from_string("(S (NP I/PRP) (VP love/VBP NLP/NNP))")
tree.leaves()                     # → ['I/PRP', 'love/VBP', 'NLP/NNP']
tree.productions()                # → ['S -> NP VP', 'NP -> I/PRP', 'VP -> love/VBP NLP/NNP']

From source

git clone https://github.com/wyattferguson/fastnltk
cd fastnltk
pip install maturin
maturin develop --release

Development

pip install -e ".[dev]"
maturin develop --release

cargo test          # 309 Rust tests
pytest tests/       # 375 Python tests (87 drop-in compat, 288 integration/unit/edge)

cargo fmt --all -- --check
cargo clippy --lib
ruff check fastnltk/ tests/

See CONTRIBUTING.md for full setup and PR workflow.

Compatibility

The goal is 100% drop-in. Right now 87 of 118 drop-in tests pass, 18 skip (no numpy installed, optional features), and 13 are marked expected-fail. Zero unexpected failures.

The 13 xfails are:

Earley parse tree extraction — Rust Earley finds parses but the tree structure differs from NLTK's chart-printing format
ConditionalFreqDist clone semantics — freqdist() returns a copy, so mutations don't propagate back the way NLTK's reference-sharing does
BigramAssocMeasures — NLTK 3.10's student_t/chi_sq internal math has edge-case behavior we match at the scoring level but not in repr
AffixTagger on untrained model — Rust backend needs training data to infer tagset
Punkt quote-start sentence detection — NLTK treats " + capital as sentence boundary; Rust doesn't implement that heuristic

These are all isolated edge cases. Every critical-path API (tokenize, tag, stem, metrics, prob, parse, chunk) is verified identically to NLTK.

Platform

Platform	Arch	Wheel
Linux	x86_64, aarch64	✅
macOS	x86_64, arm64	✅
Windows	x64	✅

License

Apache 2.0. Not affiliated with NLTK or its maintainers.

Contact + Support

Created by Wyatt Ferguson

For any questions or comments heres how you can reach me:

:octopus: Follow me on Github @wyattferguson

:mailbox_with_mail: Email me at wyattxdev@duck.com

:tropical_drink: Follow on BlueSky @wyattf

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastnltk-0.4.1.tar.gz (280.9 kB view details)

Uploaded Jul 16, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

fastnltk-0.4.1-cp310-abi3-win_amd64.whl (1.4 MB view details)

Uploaded Jul 16, 2026 CPython 3.10+Windows x86-64

fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Jul 16, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded Jul 16, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl (1.4 MB view details)

Uploaded Jul 17, 2026 CPython 3.10+macOS 11.0+ ARM64

fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl (1.5 MB view details)

Uploaded Jul 17, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file fastnltk-0.4.1.tar.gz.

File metadata

Download URL: fastnltk-0.4.1.tar.gz
Upload date: Jul 16, 2026
Size: 280.9 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1.tar.gz
Algorithm	Hash digest
SHA256	`a90dd80b91ce241b1717f215dbb2c548d25f42e5773545a7ee8ba5748f989ebb`
MD5	`bf7eb77a061c966d2c27b897492ad5b6`
BLAKE2b-256	`61bd3cc26daaa80ae1c7f6849dfe4e96e53368ee3abe24857230d495d8f2c06b`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1.tar.gz:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1.tar.gz
- Subject digest: a90dd80b91ce241b1717f215dbb2c548d25f42e5773545a7ee8ba5748f989ebb
- Sigstore transparency entry: 2187658184
- Sigstore integration time: Jul 16, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@850b05606ba343542a2a13fdefc67f47a8e655fc
- Branch / Tag: refs/tags/v0.4.1
- Owner: https://github.com/wyattferguson
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@850b05606ba343542a2a13fdefc67f47a8e655fc
- Trigger Event: push

File details

Details for the file fastnltk-0.4.1-cp310-abi3-win_amd64.whl.

File metadata

Download URL: fastnltk-0.4.1-cp310-abi3-win_amd64.whl
Upload date: Jul 16, 2026
Size: 1.4 MB
Tags: CPython 3.10+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1-cp310-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`5e08669ee27673c5c6e74fd1234b70c432f0468dd5bf21c34ec429364269ec53`
MD5	`24dd672e3da564ea02ce4b081bf69eb8`
BLAKE2b-256	`3af56a1da0ead922995f51b4c2c7d1a23199c8a5196f2e31c74a24385eab2a34`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1-cp310-abi3-win_amd64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1-cp310-abi3-win_amd64.whl
- Subject digest: 5e08669ee27673c5c6e74fd1234b70c432f0468dd5bf21c34ec429364269ec53
- Sigstore transparency entry: 2187658207
- Sigstore integration time: Jul 16, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@850b05606ba343542a2a13fdefc67f47a8e655fc
- Branch / Tag: refs/tags/v0.4.1
- Owner: https://github.com/wyattferguson
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@850b05606ba343542a2a13fdefc67f47a8e655fc
- Trigger Event: push

File details

Details for the file fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Jul 16, 2026
Size: 1.6 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`3731ee4a994bc4801315b9c0af56b986a85c865dda4c7c20baabf7fb3795a6d5`
MD5	`9d10d7d15bed3a0d4023cb66e91df55d`
BLAKE2b-256	`7dc197cfe114df70080712c1bb892039cab10621e6e4af5b20cc6eb1a8602ca5`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 3731ee4a994bc4801315b9c0af56b986a85c865dda4c7c20baabf7fb3795a6d5
- Sigstore transparency entry: 2187658190
- Sigstore integration time: Jul 16, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@850b05606ba343542a2a13fdefc67f47a8e655fc
- Branch / Tag: refs/tags/v0.4.1
- Owner: https://github.com/wyattferguson
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@850b05606ba343542a2a13fdefc67f47a8e655fc
- Trigger Event: push

File details

Details for the file fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Jul 16, 2026
Size: 1.4 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`f0b9f8382046d554f0171da500af0496f785fc9dc17b2e41aab72ff1812e2aea`
MD5	`de20f447ae650da9601ab079c73971b1`
BLAKE2b-256	`5c3b59c7a6c0171e298ff77575e63329f6f81e85df5507d637b7e1b4a9b42730`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: f0b9f8382046d554f0171da500af0496f785fc9dc17b2e41aab72ff1812e2aea
- Sigstore transparency entry: 2187658197
- Sigstore integration time: Jul 16, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@850b05606ba343542a2a13fdefc67f47a8e655fc
- Branch / Tag: refs/tags/v0.4.1
- Owner: https://github.com/wyattferguson
- Access: private
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@850b05606ba343542a2a13fdefc67f47a8e655fc
- Trigger Event: push

File details

Details for the file fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
Upload date: Jul 17, 2026
Size: 1.4 MB
Tags: CPython 3.10+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`b503896c69724bb17aac1e4e17e87e3cea4db53cc8f7d79e8f4c8c2b01afd2e4`
MD5	`de97649fa0326566044c33459e26ce5f`
BLAKE2b-256	`64f85517d5770dfc7248077264f861c57e30a8d9b83ca78558bf1362223f5479`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
- Subject digest: b503896c69724bb17aac1e4e17e87e3cea4db53cc8f7d79e8f4c8c2b01afd2e4
- Sigstore transparency entry: 2187992688
- Sigstore integration time: Jul 17, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@6a287aeef55fe03db39020eb2151e769b2f672cb
- Branch / Tag: refs/tags/v0.4.2
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6a287aeef55fe03db39020eb2151e769b2f672cb
- Trigger Event: push

File details

Details for the file fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl
Upload date: Jul 17, 2026
Size: 1.5 MB
Tags: CPython 3.10+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`89b9a27c08b33ce8c8d5fe84ee3fcb717e889b3eb2c01b19145acbd818d57f01`
MD5	`41e8df0428e397ab5b69a7d5a1816823`
BLAKE2b-256	`46cd86eb9386534b8f1e4cc94b63bd16bd50df03b67ef6512c2e8858752e87e3`

See more details on using hashes here.

Provenance

The following attestation bundles were made for fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on wyattferguson/fastNLTK

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: fastnltk-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl
- Subject digest: 89b9a27c08b33ce8c8d5fe84ee3fcb717e889b3eb2c01b19145acbd818d57f01
- Sigstore transparency entry: 2187998924
- Sigstore integration time: Jul 17, 2026
Source repository:
- Permalink: wyattferguson/fastNLTK@0073d3e6c19de284a53a2c90a78c5c24392325eb
- Branch / Tag: refs/heads/main
- Owner: https://github.com/wyattferguson
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@0073d3e6c19de284a53a2c90a78c5c24392325eb
- Trigger Event: workflow_dispatch

Release history Release notifications | RSS feed

Aug 5, 2026 4 files

Jul 18, 2026 4 files

Jul 18, 2026 4 files

Jul 17, 2026 4 files

Jul 17, 2026 4 files

This release

0.4.1 This release

Jul 16, 2026 6 files

0.4.0

Jul 15, 2026 4 files

fastnltk 0.4.1

fastNLTK

Benchmarks

What's accelerated

Install

Usage

From source

Development

Compatibility

Platform

License

Contact + Support

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed