Adaptive OCI Chunking is an extensible Python implementation for document-aware chunk selection in Retrieval-Augmented Generation (RAG). It is inspired by Ekimetrics' adaptive-chunking repository and the paper Adaptive Chunking: Optimizing Chunking-Method Selection for RAG.

The package evaluates several chunking strategies for each document, scores them with intrinsic metrics, and selects the best candidate before indexing or generation. Oracle Cloud Infrastructure (OCI) integrations are optional: the core chunking engine runs locally, while OCI Object Storage and Generative AI can be enabled when needed.

Architecture

What is Adaptive Chunking?

No single chunking method works best for every document in a RAG pipeline. Adaptive chunking treats chunking as a selection problem: try multiple splitting strategies, score each result with intrinsic quality metrics, and choose the best candidate for the document at hand.

This repo builds on that idea as a practical toolkit. It keeps the core dependency-light, adds extra production-oriented metrics, and includes optional adapters for OCI, LangChain, and LlamaIndex.

Features

• Candidate chunkers:
  • single-document
  • fixed window with overlap
  • token window with token overlap
  • recursive split
  • sentence-aware
  • paragraph-aware
  • split-then-merge
  • section-aware
  • Markdown-aware
  • delimiter-aware
  • page-aware
  • page-index hierarchical
  • semantic lexical drift
  • regex-guided section splitting
  • HTML text extraction
  • JSON structural splitting
  • code symbol splitting
  • hybrid structure-first splitting
• Metric-guided selection using paper-aligned intrinsic metrics:
  • References Completeness (RC)
  • Intrachunk Cohesion (ICC)
  • Document Contextual Coherence (DCC)
  • Block Integrity (BI)
  • Size Compliance (SC)
• Additional practical metrics:
  • source coverage
  • overlap control
  • boundary quality
  • semantic drift
  • information density
  • redundancy
• Weighted strategy selection with explainable per-metric scores.
• LangChain TextSplitter adapter.
• LlamaIndex node conversion and parser-style adapter.
• CLI for local text/Markdown files.
• Optional OCI Object Storage loader and OCI Generative AI embedding adapter.
• Small, dependency-light core for local document chunking workflows.

Contributing

Contributions are welcome for new chunkers, metrics, examples, integrations, benchmarks, documentation, and bug fixes.

See CONTRIBUTING.md for setup instructions, PR expectations, and guidance for adding chunkers or metrics.

Maintained by Yash Shukla, focused on AI, cloud, and RAG systems.

Install

Install the latest release from PyPI:

pip install adaptive-oci-chunking

The package installs the core local chunking toolkit. Optional extras are available for OCI, the API server, and framework integrations:

pip install "adaptive-oci-chunking[oci]"
pip install "adaptive-oci-chunking[api]"
pip install "adaptive-oci-chunking[pdf]"
pip install "adaptive-oci-chunking[langchain,llama-index]"

For local development from a cloned checkout:

pip install -e ".[dev]"

With OCI support from source:

pip install -e ".[oci]"

With the API server from source:

pip install -e ".[api]"

With framework integrations from source:

pip install -e ".[langchain,llama-index]"

Quick Start

Check the installed package version:

python -c "import adaptive_chunking; print(adaptive_chunking.__version__)"

From a cloned checkout, run the bundled sample through the CLI:

adaptive-chunk chunk examples/sample.md --json

After installing from PyPI in any project, create a small Markdown file and chunk it:

printf "# Demo\nAdaptive chunking chooses a splitter per document.\n\n## Details\nChunks keep related context together.\n" > sample.md
adaptive-chunk chunk sample.md --json
adaptive-chunk chunk sample.md --strategy markdown --strategy semantic --json
adaptive-chunk strategies

Python usage:

from adaptive_chunking import AdaptiveChunker, ChunkingConfig

text = "## Introduction\nAdaptive chunking chooses a splitter per document.\n\n## Details\n..."
chunker = AdaptiveChunker(
    config=ChunkingConfig(strategies=["markdown", "token-window", "semantic"])
)
result = chunker.chunk(text, document_id="demo")

print(result.strategy_name)
for chunk in result.chunks:
    print(chunk.text)

print(result.to_json(indent=2))

PDF files

Install the PDF extra, then pass a PDF path directly to the high-level chunker. Pages are separated internally with form-feed boundaries, so the page and page-index candidate strategies retain page_index metadata when selected.

pip install "adaptive-oci-chunking[pdf]"
adaptive-chunk chunk handbook.pdf --json

from adaptive_chunking import AdaptiveChunker

result = AdaptiveChunker().chunk_file("handbook.pdf")
print(result.strategy_name, result.to_json(indent=2))

PDF extraction supports text-based PDFs. Scan- or image-only PDFs must be OCRed before chunking.

Examples

Runnable examples live in examples/:

• basic_adaptive_chunking.py: end-to-end adaptive selection with metric output.
• custom_selector.py: custom chunker list and metric weights.
• langchain_integration.py: LangChain TextSplitter usage.
• llama_index_integration.py: LlamaIndex TextNode conversion.
• oci_object_storage.py: loading source text from OCI Object Storage.

Chunker Options

from adaptive_chunking.chunkers import (
    DelimiterChunker,
    MarkdownChunker,
    PageIndexChunker,
    PageChunker,
    SectionAwareChunker,
    SemanticChunker,
    TokenWindowChunker,
)
from adaptive_chunking.selector import AdaptiveSelector
from adaptive_chunking import AdaptiveChunker

selector = AdaptiveSelector(
    chunkers=[
        MarkdownChunker(max_size=1800),
        TokenWindowChunker(chunk_tokens=240, overlap_tokens=24),
        SectionAwareChunker(max_size=1800),
        DelimiterChunker(delimiter="\n---\n"),
        PageChunker(page_delimiter="\f"),
        PageIndexChunker(page_delimiter="\f"),
        SemanticChunker(max_size=1400, similarity_threshold=0.08),
    ]
)

result = AdaptiveChunker(selector=selector).chunk(text)

PageIndexChunker is separate from PageChunker: it splits by page first, then by heading hierarchy inside each page. Chunks include page_index, heading_path, section_path, section_title, and section_instance_id metadata so repeated headings such as Overview remain tied to the correct page and section occurrence during retrieval.

Every chunk returned through AdaptiveChunker also has a section_path list for retrieval filtering, for example ["Access", "MFA"]. Lines labelled as document headers, footers, page numbers, or standard page-furniture labels are ignored as section boundaries.

Available built-in strategy names can also be discovered at runtime:

from adaptive_chunking import registry

print(registry.names())
chunker = registry.create("markdown", max_size=1200)

The current built-ins are:

code
delimiter
fixed-window
html
hybrid
json
markdown
page
page-index
paragraph
recursive
regex-section
section-aware
semantic
sentence
single
split-then-merge
token-window

Metrics

The selector ranks every candidate by a weighted average of intrinsic scores. The first five metrics follow the paper's evaluation dimensions; the additional metrics make the implementation more practical for production RAG systems where dropped text, excessive overlap, and duplicated chunks are common failure modes.

Weights can be tuned:

from adaptive_chunking.metrics import IntrinsicMetricEvaluator, MetricConfig, MetricWeights
from adaptive_chunking.selector import AdaptiveSelector

weights = MetricWeights(
    block_integrity=1.4,
    coverage=1.5,
    redundancy=0.8,
)
evaluator = IntrinsicMetricEvaluator(MetricConfig(weights=weights))
selector = AdaptiveSelector(evaluator=evaluator)

Adaptive Scoring

For each document, the selector runs every candidate chunker and evaluates the chunks it produces. Each candidate receives a normalized weighted score:

score(candidate) = sum(metric_value_i * metric_weight_i) / sum(metric_weight_i)

Where:

• metric_value_i is the metric score for a candidate, normalized from 0.0 to 1.0.
• metric_weight_i controls how important that metric is for selection.
• Higher scores are better.
• Candidates are ranked from highest score to lowest score.

For example, a domain that cares about preserving source text and section boundaries might emphasize coverage and block_integrity:

Metric	Value	Weight	Weighted value
coverage	1.00	1.50	1.50
block_integrity	0.90	1.40	1.26
redundancy	0.80	0.80	0.64

score = (1.50 + 1.26 + 0.64) / (1.50 + 1.40 + 0.80)
      = 3.40 / 3.70
      = 0.919

You can inspect every candidate, not just the winner:

from adaptive_chunking import AdaptiveChunker

result = AdaptiveChunker().chunk(text, document_id="demo")

for candidate in result.candidates:
    print(candidate.strategy_name, round(candidate.score, 3), len(candidate.chunks))
    for metric in candidate.metrics:
        print(" ", metric.name, metric.value, "weight=", metric.weight)

This makes the selection process explainable: if a chunker loses, you can see whether it dropped content, produced excessive overlap, cut through structure, or failed a size constraint.

LangChain

from langchain_core.documents import Document
from adaptive_chunking.langchain import LangChainAdaptiveTextSplitter

splitter = LangChainAdaptiveTextSplitter()
documents = splitter.split_documents([
    Document(page_content=text, metadata={"source": "policy.md"})
])

# Or load and chunk a PDF directly (requires the `pdf` extra too).
pdf_documents = splitter.split_pdf("handbook.pdf")

# Every output Document retains source metadata plus document_id, chunk_index,
# start_char, end_char, strategy_name, adaptive_score, and structure metadata.

LlamaIndex

from llama_index.core.schema import Document
from adaptive_chunking.llama_index import LlamaIndexAdaptiveParser

parser = LlamaIndexAdaptiveParser()
nodes = parser.get_nodes_from_documents([
    Document(text=text, metadata={"source": "policy.md"})
])

For a direct PDF-to-node workflow:

from adaptive_chunking.llama_index import pdf_to_llama_nodes

nodes = pdf_to_llama_nodes("handbook.pdf")

LlamaIndexAdaptiveParser is a native LlamaIndex NodeParser, so it can also be passed directly to an IngestionPipeline. Output nodes retain document metadata, adaptive diagnostics, offsets, and standard source/previous/next relationships.

OCI Usage

Copy .env.example and set the values for your tenancy and compartment. The core library does not require OCI credentials unless you instantiate an OCI adapter.

from adaptive_chunking.oci import OCIObjectStorageTextLoader

loader = OCIObjectStorageTextLoader(
    namespace="my-namespace",
    bucket_name="documents",
)
text = loader.load_text("policies/example.md")

API Server

uvicorn adaptive_chunking.api:app --reload

Then post:

curl -X POST http://127.0.0.1:8000/chunk \
  -H "Content-Type: application/json" \
  -d "{\"text\":\"# Title\nBody text\", \"document_id\":\"demo\"}"

Project Layout

src/adaptive_chunking/
  chunkers.py      # candidate splitting strategies
  metrics.py       # intrinsic metric implementations
  selector.py      # weighted adaptive strategy selection
  pipeline.py      # high-level AdaptiveChunker
  langchain.py     # optional metadata-preserving LangChain adapter
  llama_index.py   # optional native LlamaIndex NodeParser and node helpers
  oci.py           # optional OCI adapters
  api.py           # optional FastAPI app
  cli.py           # command line interface
tests/
examples/

Notes

This repo is designed as a clean, extensible foundation rather than a verbatim copy of the reference implementation. The metric implementations are practical approximations intended for engineering use and experimentation. Production RAG deployments should calibrate weights, chunk sizes, and embedding models against their document domains.

References

• Ekimetrics reference implementation: ekimetrics/adaptive-chunking
• Paper: Adaptive Chunking: Optimizing Chunking-Method Selection for RAG

Citation

If this project helps your work, please cite the original adaptive chunking paper:

@inproceedings{demoura2026adaptive,
    title={Adaptive Chunking: Optimizing Chunking-Method Selection for RAG},
    author={de Moura Junior, Paulo Roberto and Lelong, Jean and Blangero, Annabelle},
    booktitle={Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026)},
    year={2026},
    url={https://arxiv.org/abs/2603.25333},
}

License

This project is licensed under the MIT License.