Pandion

Retrieval pipelines you can inspect, step by step.

Pandion parses your documents, cleans and chunks them, embeds and stores them, then searches, reranks and answers questions. Every step is a strategy you can swap, and every step tells you what it did. Each result is cached under a hash of its settings, so the same files with the same settings are never processed twice.

Pandion is named after the osprey, which spots a fish under the water and dives to retrieve it. It is the engine of RAG Playground, a web app that shows every step of a pipeline on a PDF.

Install

Pandion needs Python 3.13.

pip install "pandion[pdfium,lancedb,local]"

The base install is small. Parsers, stores and models are extras, and a step whose extra is missing tells you which one to install:

pandion install pipeline.yaml installs the extras a pipeline file needs.

Quick start

from pandion import Pipeline

p = Pipeline(
    path="./my-index",
    parse="pdfium",
    chunk={"use": "recursive_character", "chunk_size": 400, "chunk_overlap": 40},
    embed="bge-small-en-v1.5",
    store="lancedb",
    search="hybrid_rrf",
    answer="search",
)
p.index.run("report.pdf")
print(p.ask("What does the report recommend?").text)

The embedder downloads once, on first use. After that, a second p.index.run on the same file finds every step already stored and reuses it.

What you can do

• Run one step on its own, with no pipeline and no cache: pandion.parse(pdf, use="pdfium"), pandion.clean(...), pandion.chunk(...).
• Write a pipeline three ways: as a YAML file (Pipeline.from_file("pipeline.yaml")), as a dict per step, or with classes such as from pandion.parse import Docling. The three give the same pipeline and the same cache.
• Build and update an index: p.index.run(files) indexes them, and p.index.sync(folder) adds, changes and removes files to match a folder.
• Search: p.retrieve(question) runs the rewrite, search and rerank steps and returns each step's result as well as the hits.
• Ask: p.ask(question) returns the pieces found, or an answer from a chat model that cites them.

Learn more

• The guide: every step, setting and error.
• Examples: index, sync, search, ask, and the three ways to write a pipeline.
• RAG Playground: try the same pipelines in the browser.

License

MIT. See LICENSE.