watson-lite 0.1.0

Extractive QA pipeline — no LLM, no training. BM25 + FAISS + Wikidata + cross-encoder re-ranking + roberta-base-squad2.

watson-lite

A Watson-inspired extractive QA system that runs on a laptop.
No LLM. No trained weights of your own. No paid APIs.

Install

pip install watson-lite
python -m spacy download en_core_web_sm

Usage

CLI

# Single question
watson-lite "Who designed the Eiffel Tower?"
watson-lite "Who was the 44th president of the United States?"

# Interactive mode
watson-lite

Python

from watson_lite import WatsonLite

watson = WatsonLite()
answer = watson.answer("Who designed the Eiffel Tower?")

print(answer.answer)        # "Gustave Eiffel"
print(answer.confidence)    # 0.847
print(answer.source)        # "Eiffel Tower"

Example output

$ watson-lite "Who was the 44th president of the United States?"

  ANSWER:     Barack Hussein Obama
  CONFIDENCE: 43.6%
  SOURCE:     Barack Obama
  URL:        https://en.wikipedia.org/wiki/Barack Obama

  Confidence breakdown:
    extraction_model: 0.592
    span_agreement: 0.2
    graph_corroboration: 0.0
    passage_rank_signal: 1.0

  Time: 44.60s

API

• WatsonLite — Main orchestrator. answer(question) runs the full 6-stage pipeline.
• NLPProcessor — spaCy-based question classification, NER, decomposition.
• BM25Retriever — BM25 retrieval over Wikipedia REST API.
• VectorRetriever — Dense vector retrieval (sentence-transformers + FAISS).
• WikidataGraph — Structured fact enrichment from Wikidata.
• Ranker — RRF fusion + cross-encoder re-ranking.
• ExtractiveReader — Span extraction via roberta-base-squad2.
• ConfidenceScorer — Multi-signal confidence scoring.
• Cache — SQLite3 cache for Wikipedia and Wikidata responses.

Development

git clone https://github.com/daedalus/watson-lite.git
cd watson_lite
pip install -e ".[test]"

# run tests
pytest

# format
ruff format src/ tests/

# lint + type check
prospector --with-tool ruff --with-tool mypy src/

# find unused code
vulture --min-confidence 90 src/

Architecture

User Question → NLP (spaCy) → Decomposition → Entity Extraction
  → Parallel Retrieval (BM25 + FAISS) → Graph (Wikidata)
  → RRF Fusion → Cross-Encoder Rerank → Span Extraction → Confidence Score

Models Used (all pretrained, inference only)

Model	Purpose	Size
en_core_web_sm	spaCy NLP	~12MB
all-MiniLM-L6-v2	Passage embeddings	~90MB
ms-marco-MiniLM-L-6-v2	Cross-encoder reranking	~90MB
deepset/roberta-base-squad2	Extractive span QA	~480MB

Total: ~670MB — runs CPU-only.

Data Sources

• Wikipedia REST API — Live article retrieval
• Wikidata REST API — Structured entity facts (no SPARQL)

Extending

• Add a domain corpus: Replace fetch_wikipedia_passages() with your own document loader.
• Add more graph sources: Wikidata REST API pattern is reusable.
• Offline mode: Download Wikipedia dumps and index locally with BM25 + FAISS.