mkd-retriever 0.3.0

Modular, project-agnostic RAG retrieval layer: independent retrieval strategies behind a small shared core.

mkd-retriever

A modular, project-agnostic RAG retrieval layer. Several retrieval strategies — intent classification, text-to-SQL database query, and more to come (top-N vector search, query rewriting, RAPTOR, agentic flows) — live side by side in one package without depending on one another.

The PyPI name is mkd-retriever; the import name is retrieval (import retrieval).

pip install mkd-retriever                        # core only
pip install "mkd-retriever[intent_classification]"
pip install "mkd-retriever[db_query]"
pip install "mkd-retriever[all]"                 # everything implemented so far

Why it is built this way

Building retrieval methods on top of one another is a Jenga tower: changing or removing one piece topples the rest. Instead, every method is an independent module. The hard rules (enforced, see ARCHITECTURE.md):

1. A module never imports another module. Cross-module communication is the consuming project's job, not this library's.
2. Everything shared lives in retrieval.core, and core stays small. Only minimal interfaces and common data schemas — never a module's implementation detail. The dependency arrow is always module → core.
3. Dependencies are isolated per module. Each module has its own optional-dependency group; a library one module needs never leaks into another module's environment.
4. Each module is understandable and testable on its own.
5. Intent classification only returns a label — it does not route. Mapping a label to a retrieval method is the consuming project's decision.

What's inside

retrieval/
  core/        shared interfaces + ingestion-contract schemas (small, stable)
  eval/        golden-set format + metrics + async runner (cross-cutting)
  modules/
    intent_classification/   query -> intent label   (usable)
    db_query/                query -> SQL -> rows     (usable)
    top_n/                   vector search            (planned)
    query_rewriting/         rewriting / expansion    (planned)
    raptor/                  hierarchical retrieval   (planned)
    agentic/                 multi-step flows         (planned)

retrieval.core

The whole surface every module shares:

• RetrievalResult — a single scored item, returned identically by every backend (id, score, text, metadata).
• Retriever — the one async method a backend implements: async def retrieve(query, k) -> list[RetrievalResult]. run_sync is a thin convenience for synchronous callers.
• IntentLabel (9 classes) / IntentResult.
• Ingestion-contract schemas — pydantic models for the JSON an upstream ingestion/parsing project emits (chunks, product graph, document metadata). The contract is defined by shape, not by importing the producer, so any project emitting these shapes can feed this layer. Models set extra="ignore" to stay resilient to upstream format drift.

intent_classification

A prompt-based classifier over any langchain-core BaseChatModel. It returns one IntentResult and nothing more — the label → strategy mapping is the consuming project's job.

from retrieval.modules.intent_classification import (
    LLMIntentClassifier, build_default_chat_model,
)

clf = LLMIntentClassifier(build_default_chat_model())
result = await clf.classify("de1000 fiyati ne kadar")   # -> IntentResult

In tests, inject a fake BaseChatModel instead of the factory — no network, no model download. From synchronous code, use the classify_sync(clf, query) helper — the sync mirror of run_sync for retrievers.

db_query

Retrieval by generating SQL and running it. A natural-language query is turned into a SQL string by the text2sql-engine package (a 2-stage LLM pipeline), that SQL runs against a ready-made database, and the rows come back as RetrievalResult items.

from retrieval.modules.db_query import build_default_retriever

retriever = build_default_retriever(db_path="specs.db")
results = await retriever.retrieve("operating temperature of DE9001", k=5)

Two injected seams keep it testable and dependency-isolated:

• SqlGenerator (NL → SQL) — the production impl wraps a text2sql engine hitting a remote endpoint; unit tests inject a fake returning canned SQL.
• SqlExecutor (SQL → rows) — the shipped SQLiteExecutor uses only stdlib sqlite3 and opens the database read-only by default, so a stray write in generated SQL fails at the engine level.

The score defaults to 1.0 (a SQL result is an exact match set, not a similarity ranking; ordering is the query's own ORDER BY) and is overridable. The full row is in RetrievalResult.metadata. This module connects to an existing database — it does not build one.

The pipeline is optional. Every piece is importable on its own — you are not forced through DbQueryRetriever:

from retrieval.modules.db_query import SQLiteExecutor, rows_to_results, clean_sql

# Run your own SQL, reuse only the row -> RetrievalResult mapping:
cols, rows = SQLiteExecutor("specs.db").execute("SELECT * FROM products LIMIT 5")
results = rows_to_results(cols, rows, score=0.9)   # -> list[RetrievalResult]

# ...or just clean a model's SQL output, map a single row, swap either seam.

Inference & the GPU rule

Model inference (LLM/embedding) is expected to run on a separate machine behind an OpenAI-compatible endpoint. This library only sends requests to that endpoint; it never downloads models or runs local inference. The default test run (pytest) is fully mocked and never reaches a network endpoint — tests that hit a live endpoint are marked integration and skipped by default.

Development

pip install -e ".[all,dev]"
pytest

License

MIT. See LICENSE.