fpf-chunking

FPF-guided (holon / bounded-context) semantic document chunking: parses a document's structure, uses an LLM to validate which structural boundaries make good chunk boundaries, and assembles the result into token-bounded chunks — without ever exceeding the LLM's context window on a single oversized structural unit.

Install

pip install fpf-chunking

Add [gemini] if you want to use Gemini (pip install fpf-chunking[gemini]) — Anthropic and OpenAI support is included in the base install.

Usage

Boundary validation and embeddings each take a small adapter that wraps your provider's real SDK client — see fpf_chunking.adapters for AnthropicAdapter, OpenAIAdapter, and GeminiAdapter. Mix and match providers freely; nothing in the pipeline is tied to a specific one. OpenAIAdapter implements both CompletionClient (.generate(), chat completions) and EmbeddingClient (.embed()) — use one instance per role, or the same instance for both if the same model serves them.

From a file, using Anthropic for boundary validation and OpenAI for embeddings:

from anthropic import Anthropic
from openai import OpenAI
from fpf_chunking import MarkdownLoader, chunk_document, embed
from fpf_chunking.adapters import AnthropicAdapter, OpenAIAdapter

doc = MarkdownLoader().load("path/to/doc.md")

completion_client = AnthropicAdapter(
    Anthropic(api_key="..."), model="claude-sonnet-4-5-20250929"
)
chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)

embedding_client = OpenAIAdapter(OpenAI(api_key="..."), model="text-embedding-3-small")
embedded = embed(chunks, client=embedding_client)

OpenAIAdapter also works unmodified against anything that speaks the OpenAI-compatible chat/embeddings wire format — e.g. a LiteLLM proxy in front of Gemini, Anthropic, or any other provider LiteLLM supports:

from openai import OpenAI
from fpf_chunking import chunk_document
from fpf_chunking.adapters import OpenAIAdapter

# base_url points at your LiteLLM proxy; api_key is LiteLLM's own proxy
# key (master or virtual), not a real provider credential -- LiteLLM
# holds the actual upstream key itself, server-side.
client = OpenAI(base_url="http://litellm:4000", api_key="sk-litellm-...")
completion_client = OpenAIAdapter(client, model="gemini-3.6-flash")  # LiteLLM model_list name

chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)

From content already in memory (e.g. fetched over the network by your own wrapper — no filesystem path required):

from fpf_chunking import Document, chunk_document

doc = Document(doc_id="my-doc", text=buffer_from_network)
chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)

Or run the whole pipeline on Gemini, sharing one client across both roles:

from google import genai
from fpf_chunking import chunk_document, embed
from fpf_chunking.adapters import GeminiAdapter

client = genai.Client(api_key="...")
completion_client = GeminiAdapter(client, model="gemini-2.5-flash")
embedding_client = GeminiAdapter(client, model="gemini-embedding-001")

chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)
embedded = embed(chunks, client=embedding_client)

See the module docstrings for the full API: parse_holons, extract_candidate_boundaries, split_oversized_holons, assemble_chunks, FPFBoundaryValidator, fixed_size_chunks (naive baseline), and semantic_chunks (embedding-similarity baseline) are all exported for callers who want the individual pipeline stages instead of the composed chunk_document. CompletionClient/EmbeddingClient (also exported) are the two protocols every adapter implements, if you want to write your own for another provider.