konan

Like the paper angel of the Akatsuki, konan folds your documents into precise pieces — Rust chunkers behind a Python API.

Why konan?

• 🦀 Rust core — all chunking runs in native code, no Python-loop overhead
• ⚡ Multithreaded by default — chunk_many() fans out across all cores via rayon and releases the GIL; large single documents split at paragraph breaks and fan out too
• 🌀 Real async — chunk_async / chunk_many_async are native async def, not thread-pool wrappers
• 🎯 Char-accurate offsets — text[chunk.start:chunk.end] == chunk.text, always (Python slicing semantics, emoji-safe)
• 🧠 Semantic chunking — splits on topic shifts using any OpenAI-compatible embeddings endpoint, or your own async embedder
• 🔌 Ports & adapters — the Embedder port is injectable; bring your own backend

Strategies

Installation

uv add konan        # or: pip install konan

Quickstart

from konan import RecursiveChunker

chunker = RecursiveChunker(chunk_size=1000, chunk_overlap=200)

chunks = chunker.chunk(open("moby_dick.txt").read())
print(chunks[0].text, chunks[0].start, chunks[0].end, chunks[0].hash)

Parallel — all cores, one call

# rayon work-stealing across every core, GIL released:
all_chunks = chunker.chunk_many(documents)

Async — real async def, no thread-pool wrappers

chunks = await chunker.chunk_async(text)
batches = await chunker.chunk_many_async(documents)

Markdown with breadcrumbs

from konan import MarkdownChunker

chunks = MarkdownChunker(chunk_size=800).chunk(readme_text)
# chunk text is prefixed with its heading trail: "# Guide > ## Install\n\n..."
# code fences are never split

Token-exact chunks

from konan import TokenChunker

chunker = TokenChunker(chunk_size=512, chunk_overlap=64, encoding="o200k_base")

Semantic chunking

Point it at any OpenAI-compatible /embeddings endpoint (OpenAI, vLLM, Ollama, LiteLLM, …):

from konan import OpenAIEmbedder, SemanticChunker

embedder = OpenAIEmbedder(
    base_url="https://api.openai.com/v1",
    model="text-embedding-3-small",
    api_key="sk-...",
    batch_size=128,    # texts per request
    timeout=30.0,      # request timeout, seconds
    max_retries=2,     # exponential backoff on 429/5xx/connect errors
    dimensions=512,    # optional: shorten text-embedding-3-* vectors
)
chunker = SemanticChunker(embedder=embedder, threshold=0.75)
chunks = await chunker.chunk_async(article)

Or inject your own embedder — any async callable works:

async def my_embedder(texts: list[str]) -> list[list[float]]:
    return await my_model.embed(texts)

chunker = SemanticChunker(embedder=my_embedder, percentile=95.0)
chunks = await chunker.chunk_async(article)   # async-only for Python embedders

Python embedders must return list[list[float]] — call .tolist() on numpy arrays. They are async-only: chunk()/chunk_many() raise a RuntimeError pointing you at the _async variants.

The Chunk object

chunk.text      # the chunk's text
chunk.start     # char offset into the source (Python slicing semantics)
chunk.end       # char offset, exclusive
chunk.index     # 0-based position
chunk.hash      # xxh3-64 content hash, as 16 hex digits
chunk.hash_int  # the same digest as an int, if you'd only parse the hex back

Benchmarks

Benchmarked on Apple M3 Pro (arm64), Python 3.12.10. Decimal MB/s, median of 5 runs, measured from Python (the numbers you actually get). Reproduce both tables and plots with uv run --extra bench benchmarks/bench.py; Rust-level numbers with cargo bench -p konan-core (criterion) or cargo run --release -p konan-core --example bench_min (min-of-N, which is what survives a throttling machine).

Expect ±10% between runs on a laptop that throttles — the token chunker swings up to 1.4×. These figures come from the most median-typical of 17 runs, not the best one; every row in a table is from that same run, so the comparisons hold even where the absolute numbers drift.

vs other libraries (same 1 MB document, identical configs)

Throughput per strategy (1 MB document)

Parallel scaling — rayon goes brrr

64 docs × 256 KB through RecursiveChunker:

(Pure Rust puts the same workload at ~18 GB/s — 0.9 ms of the 8 ms above. The rest is building Python Chunk objects, which now dominates: the chunking itself is no longer the bottleneck.)

Caveats, honestly:

• The sentence and token rows are multi-core; the other libraries are not. Above 128 KB (sentence) and 256 KB (token), konan splits the document at paragraph breaks and segments/encodes the pieces on all cores — a paragraph break is a mandatory boundary for both UAX #29 and the cl100k/o200k pretokenizers, so the output is unchanged. Pinned to one thread with RAYON_NUM_THREADS=1 the same 1 MB document gives 342 MB/s sentence and 72 MB/s token, with identical chunk counts. Read the table rows as wall-clock on a 12-core laptop, and those two numbers as per-core efficiency.
• recursive/token use identical configs across libraries (1000 chars / 200 overlap; cl100k, 512 / 64 — note the identical token chunk counts for the libraries with the same windowing semantics). Sentence configs are not directly comparable (konan groups by chars, chonkie by tokens) — read those rows as per-library cost, not head-to-head.
• The recursive (unicode) rows run mixed German/CJK/Cyrillic/emoji prose, off konan's ASCII fast path.
• konan tokenizes with bpe-openai (tiktoken-equivalent output, much faster encoder).

Development

uv sync                       # set up the venv (builds the extension)
uv run maturin develop --uv   # rebuild after Rust changes
cargo test --workspace        # rust unit tests
uv run pytest -q              # python integration tests

The workspace is hexagonal: crates/konan-core is pure Rust (no PyO3) with Chunker and Embedder ports; crates/konan-py adapts it to Python.

License

MIT

_{Named after Konan of the Akatsuki — the only one who could fold paper into anything. 🗞️}