sift-retrieval 0.2.0

Deterministic domain-aware retrieval engine. No ML dependencies.

Sift

Deterministic domain-aware retrieval engine.
No ML dependencies. No vectors. No embeddings. No GPU.

pip install sift-retrieval

from sift import SiftEngine

engine = SiftEngine("path/to/qa_pairs/")
result = engine.search("What is the capital of Japan?")
print(result["answer"])  # Tokyo is the capital of Japan.

Quick start

# Index your data
sift --data ./my_pairs/ --build

# Query
sift "What is the capital of France?"

# Or run the demo with built-in examples
sift --demo

Benchmarks

Dataset	Pairs	Accuracy	Avg latency	P99	RAM
Knowledge domains (678 files)	904K	100%	15ms	93ms	272MB

Why

Vector search is overkill for fact retrieval. You don't need a 768-dimensional embedding to match "capital" + "japan" → "Tokyo". You need:

1. A way to compress each Q&A pair into a small set of discriminating tokens
2. A compact index mapping tokens to locations
3. A domain classifier to disambiguate polysemy ("heart" in medicine ≠ "heart" in nature)
4. Surgical disk reads instead of loading everything into RAM

Sift is all four, in ~600 lines of Python.

How it works

Multi-point signatures. Each Q&A pair is compressed into ~10-15 tokens from the question subject + first two sentences of the answer. This gives multiple entry points for matching.

Flat array index. Token→location mappings stored as array('H') + array('I') instead of Python tuples. 12M pointers in ~70MB instead of ~1.2GB.

Domain pre-filtering. Query tokens are compared against domain keyword sets. Results from the matching domain are amplified (default ×1.5), cross-domain results suppressed (default ×0.5). Unknown-domain files get neutral weight. All multipliers are tunable per deployment — the mechanism stays the same, the coefficients depend on your data mix.

Surgical disk reads. Per-file byte-offset arrays enable O(1) seeks to any pair. Only the top-K candidates are loaded into memory (default 300, tunable); the rest stay on disk. 1.3M+ pairs indexed in ~115MB RAM.

Co-occurrence re-rank. At the final scoring step, checks whether query token pairs appear in the same answer sentence. Breaks ties between candidates that match isolated tokens.

Answer verification. Key nouns from the query must appear in the answer's first sentence. Prevents false positives where a question contains shared vocabulary but the answer is about something else entirely.

Project structure

sift/
├── __init__.py   # package init
├── cli.py        # command-line interface
├── domains.py    # domain signatures + detection
├── index.py      # flat array token index
├── engine.py     # retrieval pipeline
├── speaker_adapter.py  # drop-in for existing retrievers
└── synonyms.py   # stem-aware synonym map
examples/         # supplementary data files
tests/            # test scripts

CLI

# Query the index
sift "What is the speed of light?"

# Specify data directory
sift --data /path/to/pairs "your query"

# Rebuild index from scratch
sift --data ./pairs/ --build

# Run demo (5 built-in queries)
sift --demo

Integrations

FastAPI:

from fastapi import FastAPI
from sift import SiftEngine

app = FastAPI()
engine = SiftEngine("./data/")

@app.post("/query")
def query(q: str):
    result = engine.search(q)
    return {"answer": result["answer"], "confidence": result["confidence"]}

Odoo (coming soon):

# Replace Knowledge search with Sift — instant, accurate retrieval.

Dependencies

Python standard library only. No pip install required.

FAQ

Q: How is this different from vector search?
Vector search approximates meaning. Sift matches structural token patterns. For fact retrieval (questions with specific answers), token matching is both faster and more accurate. On a benchmark of 175 real-world queries, Sift scored 100%. Vector search with the same data scored 25%.

Q: Can I use Sift with my own data?
Yes. Place Q&A pairs as JSONL files in a directory and run sift --data ./dir --build. Each line should have {"q": "...", "a": "..."}. Sift indexes everything automatically.

Q: Does Sift need a GPU?
No. Sift runs on Python's standard library. No GPU, no CUDA, no ML runtime.

Q: How much RAM does it use?
~272MB for 904K pairs. The index uses flat arrays (not Python objects) for token→location mappings, keeping memory usage predictable.

Q: Can I deploy this as an API?
Yes. The engine has zero dependencies, so it embeds into any Python web framework (FastAPI, Flask, Django) with a single import.

Changelog

• 0.2.0 — Answer verification (key-noun check), co-occurrence re-rank bonus, domain weight tuning, top-k configurable
• 0.1.0 — Initial release: flat-array index, domain classifier, CLI, surgical disk reads

License

MIT