interstiCy

A fast Rust implementation of spaCy tokenization with Python bindings.

[!NOTE] Interested in a fuller implementation of spaCy bindings in Rust? Checkout rusTy.

Overview

interstiCy is a Rust reimplementation of spaCy's English tokenizer, exposed to Python via PyO3 and maturin. It aims to be a drop-in tokenizer replacement for spacy.blank("en") and produces matching token boundaries, whitespace flags, and character-level spans.

Current scope: English only. Multi-language support is planned, but the rules and special-case loader are English-specific until loaders for other languages are added.

API

Default entry points:

For spaCy integration: intersticy.create_tokenizer(nlp)
For direct batch use: Tokenizer.load_from_spacy().tokenize_with_spans_batch(texts)

The detailed API is the primary interface:

from intersticy import Tokenizer

tok = Tokenizer.load_from_spacy()

# Single text, returns (start_char, end_char, text, has_space_after)
for start, end, text, space in tok.tokenize_with_spans("Hello, world!"):
    print(repr(text), start, end, space)
# ('Hello', 0, 5, False)
# (',', 5, 6, True)
# ('world', 7, 12, False)
# ('!', 12, 13, False)

# Batch, releases the GIL and tokenizes across cores
results = tok.tokenize_with_spans_batch([text1, text2, ...])

# String-only batch (no offsets, less overhead)
words = tok.tokenize_batch([text1, text2, ...])

# Offsets-only batch (no per-token PyString allocation)
spans = tok.tokenize_with_offsets_batch([text1, text2, ...])
# Each tuple is (start_char, end_char, has_space_after); recover text with text[start:end]

A string-only convenience method is also available:

print(tok.tokenize("Hello, world!"))
# ['Hello', ',', 'world', '!']

For spaCy integration, wrap the tokenizer as a replacement:

import spacy
import intersticy

nlp = spacy.load("en_core_web_sm")
nlp.tokenizer = intersticy.create_tokenizer(nlp)

doc = nlp("Hello, world!")
print([t.text for t in doc])
# ['Hello', ',', 'world', '!']

Benchmarks

All timings are for tokenization only, measured against spacy.blank("en") on the same machine and text.

Workload	spaCy	interstiCy	Speedup
Repetitive English paragraph (~200 k chars, cache-heavy)	~218 ms	~17 ms	~13x
Real-world prose (Pride and Prejudice, 728 k chars)	~700 ms	~60 ms	~11.8x
128 chunks of ~50 k chars, batch vs sequential	—	—	~2.2x on 8 cores

Batch APIs release the GIL and run across all cores. Larger chunks generally scale better; end-to-end pipelines spend most of their time on tagging, parsing, and NER, so the overall speedup there is smaller than the tokenizer-only figures above.

Run the standalone benchmarks yourself:

python benchmarks/benchmark.py
python benchmarks/prototype_benchmark.py

The first script downloads the public-domain Project Gutenberg text of Pride and Prejudice and reports both single-text and batch throughput. The second script compares the different batch APIs and a Rust-only count to show where the batch scaling ceiling comes from.

Parity

interstiCy is tested against spaCy for byte-level token boundaries, text, and whitespace flags.

Latest reported run (spaCy 3.8.14):

Corpus: Project Gutenberg, Pride and Prejudice
Text length: 728,798 characters
Token count: 164,234 tokens
Mismatches: 0
Span/whitespace parity: passed

Installation

pip install intersticy

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

intersticy-0.2.5.tar.gz (47.7 kB view details)

Uploaded Aug 6, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

intersticy-0.2.5-cp39-abi3-win_amd64.whl (1.0 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+Windows x86-64

intersticy-0.2.5-cp39-abi3-win32.whl (901.4 kB view details)

Uploaded Aug 6, 2026 CPython 3.9+Windows x86

intersticy-0.2.5-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+manylinux: glibc 2.17+ x86-64

intersticy-0.2.5-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (1.3 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+manylinux: glibc 2.17+ i686

intersticy-0.2.5-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+manylinux: glibc 2.17+ ARMv7l

intersticy-0.2.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+manylinux: glibc 2.17+ ARM64

intersticy-0.2.5-cp39-abi3-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded Aug 6, 2026 CPython 3.9+macOS 11.0+ ARM64

File details

Details for the file intersticy-0.2.5.tar.gz.

File metadata

Download URL: intersticy-0.2.5.tar.gz
Upload date: Aug 6, 2026
Size: 47.7 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5.tar.gz
Algorithm	Hash digest
SHA256	`739bffe0e500daf2c763fbff9b05591122781ce8d72d0c702cde7f45c1676fbf`
MD5	`b440d3b26c54168ee1c6aa04c5d7147d`
BLAKE2b-256	`dcd77ebb81c0918b9d1a833be306450d9ce450da36ebf90906b9d0daaac161da`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-win_amd64.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-win_amd64.whl
Upload date: Aug 6, 2026
Size: 1.0 MB
Tags: CPython 3.9+, Windows x86-64
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`f34278f1aa04a1854ca0060e404d840bfd4fa5d745c3a00bf6e309b4efa33cfd`
MD5	`3850c9f227d22ddf55e652a963857c77`
BLAKE2b-256	`da50f991a3df2bfd039697ef4a8b679df908836e99937617fdcc6b213622f64b`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-win32.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-win32.whl
Upload date: Aug 6, 2026
Size: 901.4 kB
Tags: CPython 3.9+, Windows x86
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-win32.whl
Algorithm	Hash digest
SHA256	`820cff1ef73df0fd91035dbf8c47f938ccd5ffb69aab81d67af8caaa9878502d`
MD5	`5ca42c8977f0c47331f64239ac663d03`
BLAKE2b-256	`f85164616a99777d9c2ee686f697b29f5f628b5c9156e91713cf7b56ec1690b3`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 6, 2026
Size: 1.3 MB
Tags: CPython 3.9+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`be4686a55d87957cb9016461356feddc789091d8f1879db20c4ba5f77ca9a459`
MD5	`b25370104f2781f412132c1a7d1a0e28`
BLAKE2b-256	`c9582897c54bdf931f7ddd79d844ec2bea39e54a88a55d2d7febcc4b25f99770`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Upload date: Aug 6, 2026
Size: 1.3 MB
Tags: CPython 3.9+, manylinux: glibc 2.17+ i686
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm	Hash digest
SHA256	`b664decd56f574039b55ad8cdbc6cd6ca76f39ac31a3d0da59bff52dc1d5483f`
MD5	`890ec1131fabbacdbaaae24c85c4159d`
BLAKE2b-256	`591687c893733708ee2d8824235c12d29064c28781a307604708e15e38cdfb4d`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Upload date: Aug 6, 2026
Size: 1.2 MB
Tags: CPython 3.9+, manylinux: glibc 2.17+ ARMv7l
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm	Hash digest
SHA256	`4f269fa9f0d8896a8403bd270f4ed85ee06d8677b458402a4f6ae11b9f653199`
MD5	`7820f8fa92199d06b0b483e2fd757cf3`
BLAKE2b-256	`1f613c60b5720609ffad70542c8b085d8606fbdf507dcb5dd91d23694381353c`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Aug 6, 2026
Size: 1.3 MB
Tags: CPython 3.9+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`c8c3ef3854b718f7b32cdf5ab5fdd9c4864688d09f3e6b0d4c19566fc124f319`
MD5	`fa1fe3d2f2a41ab7aeef78a91eba6849`
BLAKE2b-256	`bdd7d4f79674fc0954ac61e83fb9021a2a30f85af81491113f26fd40ef8364e5`

See more details on using hashes here.

File details

Details for the file intersticy-0.2.5-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: intersticy-0.2.5-cp39-abi3-macosx_11_0_arm64.whl
Upload date: Aug 6, 2026
Size: 1.1 MB
Tags: CPython 3.9+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? No
Uploaded via: uv/0.12.2 {"installer":{"name":"uv","version":"0.12.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for intersticy-0.2.5-cp39-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`b9dc9221d799e30e626825f8830d8c08fc05f02b393c1bf44614dee14331fc1b`
MD5	`4b544b58fbdda6eb9c6824200a643241`
BLAKE2b-256	`f550178211f301be4ca8009934cdaa2660f0cd17d6032e0981fa230a727bf962`

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

0.2.5 This release

Aug 6, 2026 8 files

0.1.0

Aug 5, 2026 2 files

intersticy 0.2.5

interstiCy

Overview

API

Benchmarks

Parity

Installation

Development

License

Key dates

1 maintainer

License

Requires

Provides Extra

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

Release history Release notifications | RSS feed