openembedforge 0.1.0a1

Forge specialized retrieval models from your own data.

OpenEmbedForge turns domain corpora into auditable retrieval-training datasets and specialized retrievers for RAG.

Not another vector database. Not another RAG orchestration framework. Not just another embedding-model wrapper.

OpenEmbedForge focuses on:

corpus -> training data -> specialized retrieval model -> evaluation -> deployment

Quick Start

Install the local development package:

pip install -e ".[dev]"

Use the v0.1 Python API:

from openembedforge.corpus import WordChunker, chunk_documents, load_documents
from openembedforge.data import dataset_from_chunks
from openembedforge.mining import RandomNegativeMiner

documents = list(load_documents("./examples/data"))
chunks = chunk_documents(documents, WordChunker(max_words=64, overlap_words=8))
dataset = RandomNegativeMiner(negatives_per_example=1, seed=42).mine(dataset_from_chunks(chunks))

dataset.to_jsonl(".tmp/train.jsonl")

Or run the same MVP flow from the CLI:

openembedforge corpus prepare examples/data --output .tmp/corpus.jsonl
openembedforge dataset make .tmp/corpus.jsonl --output .tmp/train.jsonl
openembedforge train --dataset .tmp/train.jsonl --output .tmp/model
openembedforge evaluate --model .tmp/model --dataset .tmp/train.jsonl

For a raw corpus path:

openembedforge train ./documents --output ./domain-model

Installation

Runtime is currently stdlib-only. Development tooling is optional:

pip install -e ".[dev]"

Future extras are reserved for heavier stacks:

pip install "openembedforge[train]"
pip install "openembedforge[onnx]"

openembedforge[train] enables the optional Hugging Face Transformers backend:

openembedforge train \
  --dataset .tmp/train.jsonl \
  --backend transformer \
  --model-name-or-path sentence-transformers/all-MiniLM-L6-v2 \
  --output .tmp/transformer-model

The default test suite builds a tiny local Transformer model, so CI does not download models.

Architecture

The current MVP includes:

• corpus loading for .txt, .md, .json, and .jsonl
• deterministic word-window chunking
• canonical RetrievalExample and JSONL datasets
• deterministic easy-negative mining
• a dependency-free dense hashing encoder for local tests and examples
• optional Transformer encoder training with an in-batch InfoNCE objective
• Recall@K, MRR, and nDCG@10
• CLI and Python APIs

Hard-negative mining, distillation, Matryoshka, ONNX, sparse, and late-interaction implementations are future work.

Roadmap

v0.2: synthetic query generation, query quality filtering, BM25/dense hard negatives, iterative mining, full provenance.

v0.3: teacher/student distillation, domain-specialized small encoders, Matryoshka dimensions, ONNX, quantization.

v0.4: continued domain pretraining, sparse retrieval, late interaction, hybrid retrieval, vector-store adapters.

v0.5: AutoRetriever, deployment-aware strategy selection, RAG-aware evaluation, experimental scratch pretraining.

CLI

openembedforge --help
openembedforge version
openembedforge corpus prepare ./documents --output ./data/corpus.jsonl
openembedforge dataset make ./data/corpus.jsonl --output ./data/train.jsonl
openembedforge dataset inspect ./data/train.jsonl
openembedforge train --dataset ./data/train.jsonl --output ./models/domain-retriever
openembedforge evaluate --model ./models/domain-retriever --dataset ./data/train.jsonl

Evaluation

Evaluation is independent of training code and reports:

• Recall@1
• Recall@5
• Recall@10
• MRR
• nDCG@10

JSON output is available:

openembedforge evaluate --model .tmp/model --dataset .tmp/train.jsonl --json

Documentation

• Branding
• Web corpus training notebook
• Roadmap
• Research questions

Contributing

See CONTRIBUTING.md.

License

Apache-2.0.