Chunky

Chunky is a python package for intelligently chunking scientific and technical repositories. It provides a modular pipeline that powers the Nancy Brain knowledge base and MCP services, while remaining useful as a standalone library for retrieval systems that need deterministic, metadata-rich chunks.

Citation

@software{chunky,
  author       = {Amber Malpas},
  title        = {AmberLee2427/chunky: v2.0.0},
  month        = mar,
  year         = 2026,
  publisher    = {Zenodo},
  version      = {v2.0.0},
  doi          = {10.5281/zenodo.18891712},
  url          = {https://doi.org/10.5281/zenodo.18891712},
}

Highlights

• Deterministic sliding-window fallback that keeps progress even on unknown file types.
• Registry-driven architecture so language-specific chunkers can be added without touching callers.
• Rich metadata (chunk_id, line_start, line_end, character spans) ready for downstream RAG and citation tooling.
• Optional forward-merge for tiny chunks via ChunkerConfig(min_chunk_chars=...), preserving content that would otherwise be dropped by downstream filters.
• Language-aware chunkers for Python, Markdown, YAML/JSON config, plain text, Fortran, reStructuredText (.rst), notebook exports (.nb.txt), and (via Tree-sitter) C/C++/HTML/Bash.
• Batteries-included tooling: Hatchling builds, Ruff linting, pytest coverage, Sphinx docs, and automated releases to PyPI + Read the Docs.

Quick Start

from pathlib import Path

from chunky import ChunkPipeline, ChunkerConfig

pipeline = ChunkPipeline()
config = ChunkerConfig(
    max_chars=1000,
    min_chunk_chars=80,  # forward-merge tiny chunks into their successor
    lines_per_chunk=40,
    line_overlap=5,
)

chunks = pipeline.chunk_file(Path("path/to/file.py"), config=config)

for chunk in chunks[:2]:
    print(chunk.chunk_id, chunk.metadata["line_start"], chunk.metadata["line_end"])

See the v2 implementation spec and v2.1 forward-merge spec for release behavior details. The original semantic chunker design draft is retained as archival context.

Documentation lives on Read the Docs: https://chunky.readthedocs.io

Built-in Chunkers

• PythonSemanticChunker — splits modules on top-level functions/classes and groups leftover module context.
• MarkdownHeadingChunker — emits chunks per heading while keeping the optional intro section.
• JSONYamlChunker — slices configs by top-level keys/items and falls back gracefully when parsing fails.
• PlainTextChunker — groups blank-line-separated paragraphs before falling back to sliding windows.
• FortranChunker — captures subroutine/function/program blocks.
• RSTChunker — detects reStructuredText section headings and chunks by section.
• NotebookChunker — groups nb4llm notebook cells (.nb.txt) into markdown+code context chunks.
• Tree-sitter chunkers (optional extra) for C/C++, HTML, Bash, and other structural languages, with gap-filling so uncaptured lines are still emitted.
• SlidingWindowChunker — deterministic line windows with overlap when no specialised handler is available.

Chunk Identifiers

Each chunk defaults to an ID of the form <doc_id>#chunk-0000. Supply a logical document identifier via Document.metadata["doc_id"] (or override the key with ChunkerConfig.doc_id_key) and customise the suffix using ChunkerConfig.chunk_id_template (both {doc_id} and {index} are available).

Forward-Merge for Tiny Chunks

Use ChunkerConfig.min_chunk_chars to preserve small leading/trailing context by merging tiny chunks into their successor (or predecessor at end-of-file) instead of dropping them in downstream pipelines.

You can also apply this utility directly:

from chunky import merge_small_chunks

Installation

Install from PyPI:

pip install chunky-files

Or install from source using the pyproject.toml metadata:

# clone the repo (if you haven't already)
git clone https://github.com/AmberLee2427/chunky.git
cd chunky

# install the library
pip install .

For development and documentation builds, install the optional extras:

pip install -e ".[dev,docs]"

To enable Tree-sitter powered chunkers for C/C++/HTML/Bash (and other supported grammars), install:

pip install chunky-files[tree]

This extra pins tree-sitter==0.20.1 alongside the bundled tree-sitter-languages so the shipped grammar binaries load correctly.

-e performs an editable install so local changes reflect immediately. .[dev,docs] installs the tooling declared under the dev and docs extras in pyproject.toml.

Tooling

• Code style: Ruff (ruff check src tests or ruff check src tests --fix)
• Tests: Pytest (pytest --cov=chunky)
• Docs: Sphinx + MyST + Furo (sphinx-build -b html docs docs/_build/html)
• Packaging: Hatchling build backend
• Versioning: bump-my-version (driven by tags and the release workflow)

Workflows

• CI tests run on Linux, macOS, and Windows for Python 3.8 through 3.12.
• Pushing a tag that matches the form vX.Y.Z triggers the release workflow. It validates that the tag matches the version in pyproject.toml, builds the distribution, and publishes to PyPI using the PYPI_API_TOKEN secret.
• Read the Docs builds the documentation automatically for pushes to the default branch. Local builds use sphinx-build -b html docs docs/_build/html.

Release checklist:

1. Review and update CHANGELOG.md, keeping the [Unreleased] section accurate.
2. Run bump-my-version bump <part> to update version metadata and append a dated entry in the changelog.
3. Build distributions locally (rm -rf dist && python -m build) and verify metadata with python -m twine check dist/*.
4. Commit the changes and push to main.
5. Tag the commit (git tag vX.Y.Z && git push origin vX.Y.Z) to trigger the Release workflow.
6. Verify the PyPI publish job and Read the Docs build succeed.

Contributing

• Know your audience: most contributors will be scientific coders. Write docs assuming limited familiarity with packaging internals.
• Use Ruff for style checks and keep numpy-style docstrings on all non-test functions.
• Target test coverage above 70% and ensure existing CI jobs pass before opening a PR.
• In pull requests, summarise code changes, testing/validation, doc updates, and provide a brief TL;DR when the description runs long.

License

Chunky is released under the MIT License.

Glossary