py-chunks

Part of chunk-engine — one Rust engine, three byte-identical SDKs (py-chunks · js-chunks · rs-chunks). Full documentation, playground and benchmarks: chunkengine.dev

PyPI Python License

The Python binding for chunk-engine. Turn any of 36 document formats into typed, structure-aware chunks for RAG — parsing and chunking run in a compiled Rust core, not a stack of Python dependencies.

Install

pip install py-chunks

Python 3.9+, and no runtime dependencies. The Rust engine ships compiled inside the wheel: it parses PDF itself, and vendors PDFium for the one job that needs a rasteriser — rendering a scanned PDF's pages when the file carries no embedded page image of its own.

Quick start

from py_chunks import get_chunks, stream_chunks, get_markdown

# Batch — works for every supported format
chunks = get_chunks("report.pdf", mode="semantic")

for chunk in chunks:
    print(chunk["content_type"])   # "heading", "table", "semantic", …
    print(chunk["content"])
    print(chunk["metadata"])       # format- and mode-specific

# Streaming — one chunk at a time
for chunk in stream_chunks("large.pdf", mode="section"):
    handle(chunk)

# Markdown conversion
md = get_markdown("report.docx")

Every chunk is a dict with content, content_type, and metadata.

📖 Chunking modes · Supported formats · Output schema · Metadata reference

Input sources

get_chunks / stream_chunks auto-detect the source:

Source Example
Path (str / Path) get_chunks("report.pdf")
bytes / bytearray / memoryview get_chunks(data, filename="report.pdf")
File-like (BytesIO, open file) get_chunks(BytesIO(data), filename="doc.md")
FastAPI / Starlette UploadFile get_chunks(upload_file)
HTTP(S) / S3 pre-signed URL get_chunks("https://bucket.s3…/f.pdf?sig=…")

Explicit helpers are also exported — get_chunks_from_path, get_chunks_from_bytes, get_chunks_from_fileobj, get_chunks_from_upload, get_chunks_from_s3_presigned_url, plus the matching stream_chunks_from_*.

A filename is required for bytes and unnamed file objects — dispatch is by extension. get_markdown accepts paths, bytes, and file objects, but not URLs.

Signatures

get_chunks(
    source, *,
    filename: str | None = None,
    mode: str = "default",
    window_size: int = 3,           # sliding_window
    overlap: int = 1,               # sliding_window (must be < window_size)
    sentences_per_chunk: int = 3,   # sentence
    paragraphs_per_page: int = 15,  # page_aware
    list_images: bool = False,
) -> list[dict] | ChunksResult

stream_chunks(source, *, filename=None, mode="default", ...) -> Iterator[dict]

get_markdown(source, *, filename=None, list_images=False) -> str | MarkdownResult

Images

from py_chunks import get_chunks

result = get_chunks("report.docx", list_images=True)   # -> ChunksResult
result.chunks   # text chunks + content_type="image" chunks
result.images   # {"7fdc906103e95537.png": b"...", ...}  <16-hex content hash>.<ext>

get_markdown(..., list_images=True) returns a MarkdownResult with .markdown (carrying ![](hash.ext) refs) and .images. Not available on stream_chunks.

Format-specific chunkers

For parameters the unified API doesn't expose (rows_per_chunk, max_chunk_chars, sheet_names, delimiter, encoding) or for per-call timing, import the format module directly. Each returns (chunks, timing) where timing is {"rust_ms": …, "python_ms": …}:

from py_chunks.chunkers.xlsx import chunk_xlsx, stream_chunk_xlsx
from py_chunks.chunkers.csv  import chunk_csv

chunks, timing = chunk_xlsx("data.xlsx", mode="row", rows_per_chunk=5)
chunks, timing = chunk_csv("data.csv", mode="row", delimiter="\t")

Modules: pdf, docx, doc, pptx, ppt, html, md, txt, xlsx (the whole spreadsheet family), csv.

Errors

Exception When
FileNotFoundError Path doesn't exist
ValueError Unsupported extension, invalid mode/parameter, missing filename
TypeError Unsupported source type, or an async upload.read()
RuntimeError Engine-level failure (e.g. a PDF with no text layer)
NotImplementedError Unsupported streaming format/mode combination

📖 Error handling · FastAPI / Flask / Django / Litestar / aiohttp / Celery recipes

Develop

pip install maturin
maturin develop --release    # rebuild the extension after any Rust change
python -m pytest -v
python -m pylint py_chunks   # expected: 10.00/10

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.6.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ranjan_kudesia from gravatar.com ranjan_kudesia

License

MIT License (MIT)

Requires

Python >=3.9

Provides Extra

dev

Tags

chunking document pdf docx docm dotx dotm doc pptx potx potm ppsx ppsm xlsx xls xlsm xlsb ods xltx xltm csv tsv msg outlook rtf epub ebook ipynb jupyter eml mbox email mime odt odp opendocument libreoffice json jsonl ndjson markdown rust

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py_chunks-0.6.0.tar.gz (8.5 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

py_chunks-0.6.0-cp313-cp313-win_amd64.whl (5.2 MB view details)

Uploaded CPython 3.13Windows x86-64

py_chunks-0.6.0-cp313-cp313-macosx_11_0_arm64.whl (5.0 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

py_chunks-0.6.0-cp313-cp313-macosx_10_12_x86_64.whl (5.1 MB view details)

Uploaded CPython 3.13macOS 10.12+ x86-64

py_chunks-0.6.0-cp312-cp312-manylinux_2_28_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

File details

Details for the file py_chunks-0.6.0.tar.gz.

File metadata

  • Download URL: py_chunks-0.6.0.tar.gz
  • Upload date:
  • Size: 8.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for py_chunks-0.6.0.tar.gz
Algorithm Hash digest
SHA256 8e7345812eda510c1bf3a5349d20bc4a9f0424d04840b7ddc1cf674f09761760
MD5 6c3bf574c2a2fc0dd657e374e294118d
BLAKE2b-256 c34b3de08f76afb7275118a688d73a6093650b9a4d087c8cf0264f6f373700ca

See more details on using hashes here.

Provenance

The following attestation bundles were made for py_chunks-0.6.0.tar.gz:

Publisher: release.yml on RanjanKudesia/py-chunks

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file py_chunks-0.6.0-cp313-cp313-win_amd64.whl.

File metadata

  • Download URL: py_chunks-0.6.0-cp313-cp313-win_amd64.whl
  • Upload date:
  • Size: 5.2 MB
  • Tags: CPython 3.13, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for py_chunks-0.6.0-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 546417d6f2b73f238e7e470408af188805e138f3f90a918d8089e784e38e397c
MD5 716027ec15d47b2c005f544a107868b7
BLAKE2b-256 53e4400556fb77cfdcab815ec8d7a5bc2570a5929f3e9446b7a93e3c360cea56

See more details on using hashes here.

Provenance

The following attestation bundles were made for py_chunks-0.6.0-cp313-cp313-win_amd64.whl:

Publisher: release.yml on RanjanKudesia/py-chunks

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file py_chunks-0.6.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for py_chunks-0.6.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a65732690d286589c1215ed0ce68b85ad35136faa8ea7fb923e458028957d3c8
MD5 710dbe2668ca913caf1936d63a736f46
BLAKE2b-256 8bd1789821e12d869eb40587536a2d3fcc339dc2330803aedd5858e309a19805

See more details on using hashes here.

Provenance

The following attestation bundles were made for py_chunks-0.6.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release.yml on RanjanKudesia/py-chunks

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file py_chunks-0.6.0-cp313-cp313-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for py_chunks-0.6.0-cp313-cp313-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 de695d267b798889cd6461faafc6c942195f9998f50895b23e98805467abda7c
MD5 a49a67a873cd45aa9680d9aba57963a1
BLAKE2b-256 49c4973958d29973f755abad93099896ac579460efc221beaf16ed289959fafa

See more details on using hashes here.

Provenance

The following attestation bundles were made for py_chunks-0.6.0-cp313-cp313-macosx_10_12_x86_64.whl:

Publisher: release.yml on RanjanKudesia/py-chunks

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file py_chunks-0.6.0-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for py_chunks-0.6.0-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 6605ead649c03ef720c56fdada2c698393cb02d6eddf6847071541c34fa8f88b
MD5 bd28a1bb4194f72b37cafd902a5b307f
BLAKE2b-256 b23cfabe0851cb73e707b2059585a81eb0b2ea994345981042c2b215d1b4c675

See more details on using hashes here.

Provenance

The following attestation bundles were made for py_chunks-0.6.0-cp312-cp312-manylinux_2_28_x86_64.whl:

Publisher: release.yml on RanjanKudesia/py-chunks

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.6.4

6 files

0.6.3

6 files

0.6.2

6 files

0.6.1

6 files

This release

0.6.0 This release

5 files

0.5.0

5 files

0.4.7

5 files

0.4.6

5 files

0.4.5

5 files

0.4.4

5 files

0.4.3

5 files

0.4.2

5 files

0.4.1

5 files

0.4.0

5 files

0.3.1

5 files

0.3.0

5 files

0.2.4

5 files

0.2.3

5 files

0.2.2

5 files

0.2.1

5 files

0.1.6

3 files

0.1.5

3 files

0.1.0

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page