idxr 0.2.0

Model-centric, config-driven, fail-stop-retry indexing toolkit for managing your index in vector databases.

idxr: Model-Centric Indexing Story

idxr exists for teams who want a dependable, repeatable way to turn any structured dataset into a searchable vector index. Everything revolves around three pillars:

• Model-centric – you describe your world as Pydantic models, and idxr keeps schemas, partitions, and manifests aligned with those models.
• Config-driven – declarative JSON configs capture how each model should be prepared and indexed, so onboarding a new dataset is as easy as committing a config file.
• Fail-stop-retry – every stage records checkpoints, row digests, and error payloads so the pipeline halts loudly when something goes wrong and then resumes from where it stopped.

A Day in the Life of an Index

The timeline below is an example run that demonstrates how idxr accompanies a team from the first dataset drop through ongoing maintenance.

1. First launch (Create)
   You register your domain models in a registry module and run:

export MODEL_REGISTRY="my_project.registry:MODEL_REGISTRY"
idxr prepare_datasets new-config foundation --model "$MODEL_REGISTRY"

idxr scaffolds a config like:

{
  "Contract": {
    "path": "datasets/contracts.csv",
    "columns": {
      "id": "CONTRACT_ID",
      "title": "CONTRACT_TITLE",
      "summary": "DESCRIPTION"
    },
    "delimiter": ",",
    "drop_na_columns": ["summary"]
  }
}

That config is committed, reviewed, and becomes the contract between data engineers and the index.

2. Daily growth (Add records)
   New exports arrive. You rerun idxr prepare_datasets with the same config; idxr deduplicates rows using digests, appends fresh partitions, and bumps manifest timestamps. No manual cleanup, no double counting.

3. Domain expansion (Add models)
   Product introduces a SupportTicket model. You add it to the registry, run idxr prepare_datasets new-config support --model "$MODEL_REGISTRY" --models SupportTicket, and drop the resulting JSON alongside the original config. idxr keeps each model’s partitions distinct but indexed in the same collection.

4. Schema shakeups (Update models)
   If Contract gains a new field, the model registry changes first. idxr prepare_datasets notices, versions the schema, and marks older partitions as stale. When idxr vectorize runs next, it honours resume checkpoints, reindexes only what changed, and writes audit-friendly error reports for anything it had to skip.

5. Operational guardrails
   During indexing, any hard failure triggers a fail-stop. idxr writes a YAML report capturing offending rows and context so you can fix the source data, then rerun idxr vectorize --resume to continue exactly where it left off. Optional E2E sampling produces JSON snippets you can review with stakeholders before the big push.

Tools in the Box

• idxr prepare_datasets – partitions CSV/JSONL sources, heals malformed rows, maintains a manifest with digests, and generates drop plans.
• idxr vectorize – streams partitions into ChromaDB (local or cloud), enforces token budgets, compacts documents via OpenAI when needed, and exports structured error reports.
• Shared libraries – offer manifest helpers, truncation strategies, drop orchestration, and CLI utilities to wire everything together.

Why idxr?

• 🔁 Lifecycle clarity – creation, accumulation, model expansion, and schema updates follow the same playbook.
• ✍️ Single source of truth – configs live in version control, so reviews and rollbacks are trivial.
• 🛑 Predictable failure semantics – when something breaks, the pipeline stops before corrupting data and tells you exactly what needs attention.
• 🔌 Bring-your-own registry – ship configs with ECC exports today, swap to CRM logs tomorrow, all with the same toolkit.
• 📦 PyPI-ready – install via pip install idxr, call the CLIs, import the libraries, and compose your own orchestration scripts.

For deep dives and operational recipes, explore FAQ.md, DOC.md, TRUNCATION_EXAMPLES.md, and ERROR_HANDLING.md.