---

Most "AI + data" tools are fetchers: give them a URL, they hand back a string, then forget everything. The string has no date, no provenance, no dedup, no memory, and no way to verify a claim afterwards.

Anchor is the layer above the fetchers. Every piece of knowledge it acquires carries a resolvable anchor back to a human-openable source (t=1247, p.14, ¶14, bbox), a publication date, a quality score, and freshness metadata. It can tell an agent not just what it knows, but what it doesn't — and when an answer is wrong, one click shows you exactly why.

Success criterion: not "the AI stops being wrong." It's "when the AI is wrong, one click shows you exactly why."

---

⚡ Install in Claude Code (30 seconds)

Once published to PyPI, add Anchor as an MCP server with one command:

claude mcp add anchor -- uvx --from anchorx anchor-mcp

That's it — no clone, no venv, no Ollama. On first launch Anchor auto-installs its browser, downloads a tiny CPU embedding model, and starts serving these tools to Claude:

anchor_ingest · anchor_ask · anchor_search · anchor_research · anchor_coverage · anchor_fetch · anchor_refresh · anchor_clear

Then just talk to Claude:

"Ingest this PDF and these 2 articles into Anchor, then answer my question with citations."

Or from source (until the PyPI release):

git clone https://github.com/syedawais355/anchorx.git && cd anchorx
pip install -e .
python -m playwright install chromium
claude mcp add anchor -- "<path>/.venv/Scripts/anchor-mcp"

---

✨ Why Anchor

Principle	What it means	Consequence in the design
Provenance is the product	An answer is only useful if you can open the source and check it	Every chunk carries a resolvable anchor; citations deep-link to the exact spot
Pre-filter, not post-filter	"Only videos over 50k views after March 2026" must be enforced inside retrieval	Metadata filters push down into both BM25 and the vector scan
Absence is information	The model must know the scope of what it has	coverage() is a first-class tool that reports gaps, not a debug endpoint
Adapters are commodity	Fetching is a solved, high-maintenance problem	Thin adapters wrap existing libraries; swap one in an afternoon, nothing else changes

---

🏗️ Architecture

Read it bottom-up — the durable value lives at L0/L1; the adapters at L2 are replaceable parts.

┌─────────────────────────────────────────────────────────────────────┐
│  L4  INTERFACE        MCP server  │  CLI  │  Python SDK              │
│      ask · search · ingest · fetch · coverage · refresh · research   │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L3  ORCHESTRATION    Budget Governor · Job Queue · Research · Sync   │
└────────────────────────────────┬────────────────────────────────────┘
        ┌────────────────────────┴───────────────────────┐
┌───────▼──────────────────────────┐   ┌─────────────────▼────────────┐
│  L2  ACQUISITION (Adapters)       │   │  L2' RETRIEVAL               │
│  youtube · web_deep · web_search  │   │  pre-filter → BM25 + dense   │
│  documents · images               │   │  → RRF → cross-encoder rerank│
└───────────────┬───────────────────┘   └─────────────────┬────────────┘
┌───────────────▼─────────────────────────────────────────▼────────────┐
│  L1  NORMALIZATION    RawPayload → Document → Chunks → Vectors        │
│      provenance stamping · anchor assignment · quality scoring        │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L0  CORPUS    SQLite (metadata · FTS5 · graph) · LanceDB (vectors)  │
│      blob cache · dedup · freshness · coverage · grounding           │
└─────────────────────────────────────────────────────────────────────┘

---

🔄 How it works

Ingestion — from a URL to a grounded, chunked Document

flowchart LR
    U["URL / file"] --> R{Router}
    R --> L["Adapter ladder<br/>(escalate only when needed)"]
    L --> N["Normalize → Document<br/>provenance + anchors"]
    N --> C["Structure-aware<br/>chunking"]
    C --> E["Embed<br/>(bge-m3 / offline)"]
    E --> S[("SQLite + LanceDB<br/>+ blob cache")]

Query — filter first, then fuse, then rerank, then ground

flowchart LR
    Q["Question"] --> F["Metadata<br/>pre-filter"]
    F --> B["BM25<br/>(FTS5)"]
    F --> D["Dense<br/>(LanceDB)"]
    B --> X["RRF fusion<br/>k=60"]
    D --> X
    X --> RR["Cross-encoder<br/>rerank → top 8"]
    RR --> A["Answer + citations<br/>+ grounding report"]

research() — the multi-round acquisition loop (capped at depth 3)

flowchart TD
    A[Question] --> B[Discover candidates]
    B --> C[Score: authority × recency × relevance / cost]
    C --> D{Within budget?}
    D -->|no| G[Trim to top-N] --> E
    D -->|yes| E[Acquire → normalize → persist]
    E --> H[Retrieve + rerank]
    H --> I[Coverage check]
    I -->|gaps found, depth left| J[Refine queries] --> B
    I -->|sufficient| K[Answer + citations + grounding]

---

⚓ Anchors — what makes citation real

Every chunk resolves back to a human-openable location. This is the difference between "RAG" and verifiable RAG.

Source	Anchor payload	Renders as	Opens to
youtube	{"t": 1247}	[20:47]	youtu.be/ID?t=1247
web	{"sel": "…", "para": 14}	¶14	URL + scroll-to
document	{"page": 14, "bbox": […]}	p.14	PDF page + highlight
image	{"region": […]}	region	image + box overlay
search_result	{"rank": 3}	SERP #3	original URL

---

🔌 Adapters & acquisition ladders

Adapters escalate through tiers only when needed — a static GET costs ~200ms; a browser costs seconds.

Adapter	Ladder
web_deep	Tier Method Notes L0 httpx + trafilatura static fetch (~200ms) L1 needs_js() classifier detect SPA shells L1.5 __NEXT_DATA__ / JSON-LD zero-browser content L2 Playwright Chromium JS render + stability wait L3 + stealth / proxy / cookies opt-in only
youtube	Tier Method Quality T0 manual captions 0.95 T1 auto captions 0.70 T2 faster-whisper + VAD 0.90 T3 + pyannote diarization 0.92
documents	Tier Method P0/P1 pypdfium2 text layer P3 OCR (injectable) — DOCX · PPTX · EPUB → Markdown — CSV · XLSX → Data Card (never chunked as prose)
images	pHash dedup → classify → route → VLM (chart→table, diagram, caption) + OCR. Embedded images become child Documents linked to their parent page.
web_search	Discovery only (nothing persisted unless asked): SearXNG · Brave · Exa, with a snippet-sufficiency check.

---

📊 Coverage & grounding — the differentiator

coverage(topic, filters) tells you what the corpus holds and where it's thin:

{
  "n_documents": 47,
  "by_source": { "youtube": 12, "web": 28, "document": 5, "image": 2 },
  "date_histogram": { "2024": 31, "2025": 14, "2026": 2 },
  "domain_concentration": 0.62,
  "authority": { "primary": 4, "secondary": 31, "unknown": 12 },
  "gaps": [
    "No source newer than 2025-12 — topic likely evolved",
    "62% of coverage from a single domain — low independence",
    "No primary/official documentation present"
  ]
}

Every ask() returns a grounding report — including unsupported_spans, the sentences that could not be entailed by the retrieved evidence:

{
  "grounding": {
    "chunks_used": 6, "sources_used": 4, "independent_domains": 3,
    "oldest_evidence": "2024-08-11", "newest_evidence": "2026-03-11",
    "unsupported_spans": ["Penguins architected the platform overnight."],
    "warnings": ["2 of 4 sources are stale (>180d)"]
  }
}

---

🎯 Quality scoring

A single quality_score ∈ [0,1] per document — components stored separately so scores can be re-weighted without re-ingesting.

quality = 0.30 × acquisition_fidelity      # tier: T0=.95 T1=.70 T2=.90 · L0=.90 · P1=.95
        + 0.25 × extraction_completeness    # text ratio · table survival · no truncation
        + 0.20 × authority                  # primary source · domain · named author
        + 0.15 × recency_fit                # half-life decay vs. topic volatility
        + 0.10 × structure                  # headings · resolved sections · valid anchors

---

📦 Installation

Requires Python 3.12+.

git clone https://github.com/syedawais355/anchorx.git
cd Anchor-AI

python -m venv .venv
.venv\Scripts\Activate.ps1        # Windows (PowerShell)
# source .venv/bin/activate       # macOS / Linux

pip install -e ".[dev]"
python -m playwright install chromium   # for web_deep L2/L3

Optional model backends (each has an offline fallback, so nothing here is required to run):

pip install -e ".[asr]"          # faster-whisper (YouTube T2)
pip install -e ".[diarization]"  # pyannote.audio (YouTube T3)
pip install -e ".[rerank]"       # sentence-transformers cross-encoder
pip install -e ".[clip]"         # open-clip text↔image search
pip install -e ".[ocr]"          # pytesseract
pip install -e ".[stealth]"      # playwright-stealth (web_deep L3)

---

🚀 Usage

CLI

anchor ingest https://example.com/post https://youtu.be/VIDEO_ID
anchor ask "how does structure-aware chunking work?" --min-quality 0.6 --after 2026-01-01
anchor search "provenance"

anchor ask runs entirely offline against the corpus — fast, free, and network-free.

MCP server

anchor-mcp     # exposes anchor_ask / coverage / search / ingest / fetch / refresh

Python

from anchor_ai.adapters.web_deep import acquire, render_page
from anchor_ai.corpus import DocumentStore, VectorStore, HashingEmbedder, coverage

doc = acquire("https://example.com/post", render=render_page)  # L0→L2 ladder
print(doc.title, doc.quality_score, [s.heading for s in doc.sections])

report = coverage(store.conn, "chunking")
print(report.gaps)

---

🗂️ Project structure

anchor_ai/
├── config.py                   # resolved storage paths
├── core/                       # L1 — the keystone
│   ├── document.py             # Document · Chunk · Section · Anchor · provenance
│   ├── protocols.py            # SourceAdapter · Filters · Candidate · CostEstimate
│   ├── chunking.py             # structure-aware splitter (never splits code/tables)
│   └── quality.py              # 5-part weighted scoring
├── adapters/                   # L2 — replaceable acquisition
│   ├── youtube/                # discover · acquire (captions) · asr · diarization · postprocess
│   ├── web_deep/               # static · detect · embedded · browser · extract (ladder)
│   ├── web_search/             # searxng · brave · exa
│   ├── documents/              # pdf · office (docx/pptx/epub) · tabular (data cards)
│   └── images/                 # phash · classify · vlm · ocr · asset linking
├── corpus/                     # L0 — durable value
│   ├── store.py                # SQLite + FTS5 + migrations + DocumentStore
│   ├── vectors.py              # LanceDB (native metadata pre-filter)
│   ├── embeddings.py           # Ollama + offline embedders
│   ├── retrieval.py            # filter → BM25 + dense → RRF → fuse
│   ├── rerank.py               # cross-encoder reranker
│   ├── blobs.py                # content-addressed raw-payload cache
│   ├── dedup.py                # MinHash near-duplicate + corroboration edges
│   ├── freshness.py            # per-source TTLs + refresh
│   ├── coverage.py             # corpus composition + gaps
│   ├── grounding.py            # citations + grounding report
│   ├── entailment.py           # unsupported-span detection
│   └── image_index.py          # CLIP text↔image index
├── orchestration/              # L3
│   ├── budget.py               # cost governor + graceful degradation
│   ├── queue.py                # resumable SQLite job queue
│   ├── research.py             # multi-round research loop
│   └── sync.py                 # incremental channel/site sync
├── interfaces/                 # L4
│   ├── cli.py                  # anchor CLI
│   └── mcp_server.py           # anchor-mcp server
└── tests/                      # 387 tests

---

🧰 Tech stack

Concern	Choice	Why
Metadata store	SQLite + FTS5	one file, BM25 included, transactional
Vectors	LanceDB	embedded, native metadata pre-filtering
Static extract	trafilatura	best boilerplate removal per CPU cycle
Browser	Playwright	superior waiting primitives + context pooling
YouTube	yt-dlp	the only thing that keeps working
ASR	faster-whisper + VAD	4× faster, word timestamps, kills hallucination
PDF	pypdfium2	fast text layer, escalate only when needed
MCP	fastmcp	least ceremony

---

🧪 Testing

pytest            # 387 tests — every adapter, ladder, and pipeline

Heavy/optional backends are behind lazy imports with deterministic offline fallbacks, so the full suite runs green with no model servers — while real Chromium renders, PDF/office round-trips, and vector search are genuinely exercised.

---

🛣️ Build phases

Phase	Scope	Status
P1	Schema · corpus · chunking · hybrid retrieval · youtube · web_deep · CLI	✅
P2	MCP server · pre-filters · coverage() · grounding · dedup · freshness	✅
P3	documents (PDF/office/data cards) · web_search · job queue · budget governor	✅
P4	images (VLM+OCR) · embedded-asset linking · research() · entailment	✅
P5	CLIP index · diarization · stealth render · incremental sync	✅

---

📄 License

Released under the MIT License — see LICENSE for the full text.

MIT License

Copyright (c) 2026 Syed Muhammad Awais Gillani

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.