omni-ingest 0.1.0

OmniIngest is a multimodal ingestion framework for building reusable knowledge pipelines. It turns documents, web pages, images, and audio into validated knowledge items without the faff of stitching together one-off preprocessing scripts.

Documentation: omniingest.onrender.com

Why use OmniIngest

• Multimodal processing. PDF, DOCX, HTML, images, audio, and plain text use one pipeline model.
• Multiple model providers. Every model-backed step can select its own Pydantic AI model. Chat and embedding models are configured independently.
• Storage-independent pipelines. YAML describes transformation. CLI or Python caller chooses JSON, vector store, graph store, or another output.
• Structured when needed. JSON Schema output, jq transforms, and step validation turn model results into predictable data.
• Built for sizeable jobs. fsspec resources, concurrent stages, progress events, and stage checkpoints keep long ingestion runs manageable.

Quick start

Install command-line tool with uv:

uv tool install omni-ingest

Create a pipeline which extracts text and makes sentence-aware chunks:

pipeline_id: school_notes
domain_profile: education

steps:
  - agent: text
  - agent: chunking
    config:
      chunk_size: 900
      overlap: 120

Run it against a PDF:

omni-ingest school_notes.yaml \
  --input lessons/soil-erosion.pdf \
  --output build/soil-erosion.json

YAML stops at transformation. Pick storage at run time:

omni-ingest school_notes.yaml --input lessons/soil-erosion.pdf --output - | jq '.items | length'
omni-ingest retrieval.yaml --input handbooks/*.pdf --output vector:default

Learn more

• Command-line usage covers files, remote resources, stdin, outputs, and resume.
• Pipeline playbook builds from simple YAML to structured output, validation, components, and model tools.
• Working with event stream covers progress and live application integration.
• API reference documents Python interfaces.

Development

git clone https://github.com/A4i-tech/OmniIngest.git
cd OmniIngest
uv sync --dev
uv run pytest tests/unit