Multimodal ingestion that does not dictate your stack
OmniIngest is a multimodal ingestion framework for building reusable knowledge pipelines. It turns documents, web pages, images, and audio into validated knowledge items without the faff of stitching together one-off preprocessing scripts.
Documentation: omniingest.onrender.com
Why use OmniIngest
- Multimodal processing. PDF, DOCX, HTML, images, audio, and plain text use one pipeline model.
- Multiple model providers. Every model-backed step can select its own Pydantic AI model. Chat and embedding models are configured independently.
- Storage-independent pipelines. YAML describes transformation. CLI or Python caller chooses JSON, vector store, graph store, or another output.
- Structured when needed. JSON Schema output, jq transforms, and step validation turn model results into predictable data.
- Built for sizeable jobs. fsspec resources, concurrent stages, progress events, and stage checkpoints keep long ingestion runs manageable.
Quick start
Install command-line tool with uv:
uv tool install omni-ingest
Create a pipeline which extracts text and makes sentence-aware chunks:
pipeline_id: school_notes
domain_profile: education
steps:
- agent: text
- agent: chunking
config:
chunk_size: 900
overlap: 120
Run it against a PDF:
omni-ingest school_notes.yaml \
--input lessons/soil-erosion.pdf \
--output build/soil-erosion.json
YAML stops at transformation. Pick storage at run time:
omni-ingest school_notes.yaml --input lessons/soil-erosion.pdf --output - | jq '.items | length'
omni-ingest retrieval.yaml --input handbooks/*.pdf --output vector:default
Learn more
- Command-line usage covers files, remote resources, stdin, outputs, and resume.
- Pipeline playbook builds from simple YAML to structured output, validation, components, and model tools.
- Working with event stream covers progress and live application integration.
- API reference documents Python interfaces.
Development
git clone https://github.com/A4i-tech/OmniIngest.git
cd OmniIngest
uv sync --dev
uv run pytest tests/unit
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file omni_ingest-0.1.1.tar.gz.
File metadata
- Download URL: omni_ingest-0.1.1.tar.gz
- Upload date:
- Size: 46.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
efee288595e019554017f74319c4d0c9b3ef9b6902cd64c1ba1bee4422a9d899
|
|
| MD5 |
4d84720c707c490f16878e57f891fc9e
|
|
| BLAKE2b-256 |
72a0faa8f44bfaa66872ad8bb899b74b6bcef676290ea89b9716ce3cb6dd89ed
|
Provenance
The following attestation bundles were made for omni_ingest-0.1.1.tar.gz:
Publisher:
main.yaml on A4i-tech/OmniIngest
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
omni_ingest-0.1.1.tar.gz -
Subject digest:
efee288595e019554017f74319c4d0c9b3ef9b6902cd64c1ba1bee4422a9d899 - Sigstore transparency entry: 2468133143
- Sigstore integration time:
-
Permalink:
A4i-tech/OmniIngest@f355d2910b05f748ae66846fb8d886f84f8ce961 -
Branch / Tag:
refs/tags/v0.1.1 - Owner: https://github.com/A4i-tech
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
main.yaml@f355d2910b05f748ae66846fb8d886f84f8ce961 -
Trigger Event:
push
-
Statement type:
File details
Details for the file omni_ingest-0.1.1-py3-none-any.whl.
File metadata
- Download URL: omni_ingest-0.1.1-py3-none-any.whl
- Upload date:
- Size: 59.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2add91cd7dcdb27ff0b1f97200cd0ffd4e512449b2fad865a3227371647c6827
|
|
| MD5 |
cb9373ef74a14ccadcd6d3c7c07314e7
|
|
| BLAKE2b-256 |
b97fb837cf5fd5053c72836bc36f8b77031169ebabcd547225ae3a496f5c192e
|
Provenance
The following attestation bundles were made for omni_ingest-0.1.1-py3-none-any.whl:
Publisher:
main.yaml on A4i-tech/OmniIngest
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
omni_ingest-0.1.1-py3-none-any.whl -
Subject digest:
2add91cd7dcdb27ff0b1f97200cd0ffd4e512449b2fad865a3227371647c6827 - Sigstore transparency entry: 2468133262
- Sigstore integration time:
-
Permalink:
A4i-tech/OmniIngest@f355d2910b05f748ae66846fb8d886f84f8ce961 -
Branch / Tag:
refs/tags/v0.1.1 - Owner: https://github.com/A4i-tech
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
main.yaml@f355d2910b05f748ae66846fb8d886f84f8ce961 -
Trigger Event:
push
-
Statement type:
