chunking-ai 0.0.1

A fast and light-weight library for ingesting and chunking files

Installation

pip install ...

Install the following system dependencies for the chunking library:

• pandoc: for parsing markup language (e.g. epub, html, rtf, rst, docx...)
• libreoffice: for file conversion (e.g. doc, ppt, xls to docx, pptx, xlsx...)

Features

Library responsibility:

Graph: Files ----> Parse ----> Chunk ----> Index ----> Search ----> Represent retrieval context -----> LLM We are responsible for:

• Parse, Chunk
• Represent retrieval context

How chunking is different? We chunk the document structure rather than the text. This is to take the document structure into account to create more sensible chunks.

• Maintain sectional layout structure during parsing and chunking (*).
• Supported formats (refer ... for suitable parsers for each format):
  • Text: html, md, txt, epub, latex, org, rtf, rst
  • Office documents: pdf, docx, pptx, xlsx
  • Images: jpg, png
  • Audio: wav, mp3
  • Video: mp4
  • Code: ipynb, (coming soon: py, js)
  • Data interchange: csv, json yaml, toml
• Content linking across files.
• Suppport LLM integration for correct content parsing.
• Task description for agent-oriented RAG strategies.
• Fast.
  • At least 100MB/s parsing
  • At least 100MB/s splitting
• Extensible.
  • Easy to add new strategy
  • Easy to change configure of the current strategy
• Traceable: trace from chunk to source.
• Developer-friendly
  • Evaluation
  • Benchmark
  • Config selector
  • Docker
  • CLI to chunk
• Complete.
  • All common file types
  • All chunking strategies

(*) Due to the complexity and variety of how structures can be represented in different file types, there can be errors. chunking treats this as best effort. Refer xxx for difficult cases. File an issue if you encounter a problem.

Usage

Add LLM support

By default, chunking uses the llm (repo) with alias chunking-llm to interact with LLM. Please setup the desired LLM provider according to their docs, and set the alias chunking-llm to that model. Example, using Gemini model (as of April 2025):

# Install the LLM gemini
$ llm install llm-gemini

# Set the Gemini API key
$ llm keys set gemini

# Alias LLM to 'chunking-llm' (you can see other model ids by running `llm models`)
$ llm aliases set chunking-llm gemini-2.5-flash-preview-04-17

# Check the LLM is working correctly
$ llm -m chunking-llm "Explain quantum mechanics in 100 words"

Cookbook

TBD.

• Application in agent-oriented RAG strategies.

Examples

TBD. Code snippet of prominent features:

• Drop-in replacement for file parsing.
• Show case of maintaining sectional layout structure. Show case the Chunk interface.
• Show case of a notable chunking strategy.
• Use as tool for agent.

Contributing

Ensure that you have git and git-lfs installed. git will be used for version control and git-lfs will be used for test data.

# Clone the repository
git clone [e-mail]:chunking-ai/chunking.git
cd chunking

# Fetch the test data
git submodule update --init --recursive

# Install development dependnecy
pip install -e ".[dev]"

# Initialize pre-commit hooks
pre-commit install

License

Apache 2.0.