embedding-ingestion 1.0.0

No project description provided

embedding-ingestion

embedding-ingestion is a packaged ingestion pipeline that loads a text dataset from MinIO, generates embeddings through an OpenAI-compatible embedding endpoint, and stores vectors in Qdrant.

The project is intentionally config-driven. The runner, loader, dataset, embedder, and processor classes are all resolved from module_path values, which makes the package reusable across datasets and model backends without changing application code.

What it does

Current built-in pipeline:

1. Load a TextDataset subclass from MinIO.
2. Convert dataset rows into LangChain Document objects.
3. Generate embeddings asynchronously.
4. Recreate a Qdrant collection.
5. Upsert document vectors and metadata into Qdrant.
6. Verify that points were written successfully.

Core components:

• MinioLoader in src/embedding_ingestion/loaders.py
• QdrantStore in src/embedding_ingestion/store.py
• MinioVLLMQdrantRunner in src/embedding_ingestion/runners.py
• CLI entrypoint in src/embedding_ingestion/main.py

Architecture

The package defines three extension points:

• DocumentLoader: responsible for fetching and optionally filtering documents.
• Store: responsible for embedding, persistence, and post-write verification.
• Runner: orchestrates the end-to-end ingestion flow.

At runtime, the CLI resolves the runner from the YAML config file at /config/config.yaml, then instantiates nested loader/store settings through Pydantic models.

Requirements

• Python >=3.11,<3.13
• A reachable MinIO-compatible object store
• A reachable Qdrant instance
• An OpenAI-compatible embeddings endpoint
• A dataset class that subclasses retrievalbase.dataset.TextDataset

Installation

Local

Production dependencies:

make install

Development environment:

make dev-install

If you prefer using uv directly:

uv sync --group dev --all-extras

Configuration

The application expects a YAML config at /config/config.yaml by default.

Example:

module_path: embedding_ingestion.runners.MinioVLLMQdrantRunner

loader:
  module_path: embedding_ingestion.loaders.MinioLoader
  endpoint: minio:9000
  access_key: ${MINIO_ACCESS_KEY}
  secret_key: ${MINIO_SECRET_KEY}
  bucket: datasets
  dataset_module_path: your_project.datasets.MyTextDataset
  dataset_minio_path: corpora/my-dataset.parquet

store:
  module_path: embedding_ingestion.store.QdrantStore
  url: http://qdrant:6333
  collection_name: my_embeddings
  distance: cosine
  embedder:
    module_path: retrievalbase.evaluation.openai_compatible.OpenAICompatibleEmbedder
    model_name: text-embedding-3-large
    base_url: http://vllm:8000/v1
  processor:
    module_path: retrievalbase.evaluation.nomic.NomicProcessor

Running

CLI

After the config file is mounted or created at /config/config.yaml:

embedding-ingestion

Equivalent:

python -m embedding_ingestion.main

Docker

Build:

docker build -t embedding-ingestion .

Run:

docker run --rm \
  -v /absolute/path/to/config:/config \
  embedding-ingestion

The image entrypoint runs:

python -m embedding_ingestion.main

Best practices

Treat ingestion as destructive by default

QdrantStore.create() deletes and recreates the target collection before writing. Use a dedicated collection per run or environment, and do not point this job at a production collection unless full replacement is intended.

Use stable dataset classes

dataset_module_path must point to a concrete TextDataset subclass with a working from_minio(...) implementation. Keep that class in a versioned package so ingest behavior remains reproducible.

Validate non-app dependencies before running

Before executing the pipeline, confirm:

• the MinIO bucket and object key exist
• the Qdrant URL is reachable
• the embedding endpoint is healthy
• the selected embedding model returns the expected vector dimension

Watch for deduplication behavior

The built-in loader groups rows by page_content and keeps the first metadata entry. If duplicate text with different metadata matters in your use case, change that behavior before relying on the default loader.

Make runs idempotent where possible

Point IDs are generated deterministically from page_content and metadata. That is good for stable re-ingestion semantics, but only if your upstream dataset normalization is also stable.

Extending the package

Add a custom implementation when you need a different source or vector store:

• subclass DocumentLoader for a new ingestion source
• subclass Store for a new destination
• subclass Runner if orchestration needs to change

Then point the YAML module_path values at your custom classes.

Repository layout

src/embedding_ingestion/
  __init__.py      # abstract loader/store/runner contracts
  loaders.py       # MinIO dataset loader
  store.py         # Qdrant embedding store
  runners.py       # concrete pipeline runner
  settings.py      # config models
  utils.py         # runner loading and deterministic document IDs
  main.py          # CLI entrypoint