fpf-chunking

FPF-guided (holon / bounded-context) semantic document chunking: parses a document's structure, uses an LLM to validate which structural boundaries make good chunk boundaries, and assembles the result into token-bounded chunks — without ever exceeding the LLM's context window on a single oversized structural unit.

Install

pip install fpf-chunking

Usage

From a file:

from anthropic import Anthropic
from fpf_chunking import MarkdownLoader, chunk_document

doc = MarkdownLoader().load("path/to/doc.md")
chunks = chunk_document(
    doc,
    anthropic_client=Anthropic(api_key="..."),
    anthropic_model="claude-sonnet-4-5-20250929",
    max_chunk_tokens=400,
)

From content already in memory (e.g. fetched over the network by your own wrapper — no filesystem path required):

from fpf_chunking import Document, chunk_document

doc = Document(doc_id="my-doc", text=buffer_from_network)
chunks = chunk_document(doc, anthropic_client=..., anthropic_model=..., max_chunk_tokens=400)

Then embed however you like:

from openai import OpenAI
from fpf_chunking import embed

embedded = embed(chunks, client=OpenAI(api_key="..."), model="text-embedding-3-small")

See the module docstrings for the full API: parse_holons, extract_candidate_boundaries, split_oversized_holons, assemble_chunks, FPFBoundaryValidator, fixed_size_chunks (naive baseline), and semantic_chunks (embedding-similarity baseline) are all exported for callers who want the individual pipeline stages instead of the composed chunk_document.