RAGSpine

The framework-free backbone for backend RAG. Deterministic dual-channel retrieval and agent orchestration, with anti-fabrication and source provenance built in — no Dify, no LangGraph, no DSL. Just composable Python.

Spine 家族 / Spine family

本仓库是 Spine 家族的成员之一（角色：L1 引擎）。家族全部成员、分层、依赖方向、依赖形式与当前差距见 docs/spine-family.md；该文件在每个家族仓库中的副本内容相同，真源在家族根目录 ~/startup/spine/docs/spine-family.md，用根目录 make family-doc-sync 同步。

🤖 给 AI / LLM: 用本库前先读 llms.txt（精简索引）与 docs/llms/（完整 API / recipes / 陷阱）；pip install 后这些文档随包位于 site-packages/ragspine/_llms/。

What is RAGSpine?

RAGSpine is a backend RAG engine you assemble in plain Python — not a framework you submit to. Most stacks force a choice between hand-rolled glue and heavyweight orchestration platforms (Dify, LangGraph) that drag in their own runtime, graph DSL, UI, and lock-in. RAGSpine is the middle path: a coherent, batteries-included library of composable parts — retrieval, agent orchestration, document extraction, evaluation, and an HTTP service layer — wired together by ordinary functions and typed Protocols.

It was built for a demanding use case (executive insight Q&A over financial and operational reports) and ships that rigor as first-class, code-enforced invariants:

• Never fabricate. When the data isn't there, the orchestrator deterministically refuses ("not found") regardless of what the LLM says. Anti-fabrication lives in the control flow, not in a prompt you hope the model obeys.
• Always cite. Every answer carries source lineage (document + locator).
• Two channels, one router. A deterministic structured/numeric channel (a fact table + function-calling) answers "what's the number," and a narrative RAG channel (hybrid retrieval + rerank) answers "why / what happened." An agent routes each question — or splits a composite one and runs both.
• Everything is pluggable. LLM provider, embedding backend, reranker, OCR, retriever, task queue — all are typed Protocols injected at the edges. The core imports zero SDKs and runs fully offline with a deterministic MockProvider.

Why RAGSpine

RAG capabilities

RAGSpine covers the mainstream RAG technique surface — benchmarked stage-by-stage against LlamaIndex · LangChain/LangGraph · Haystack · RAGFlow · Weaviate · Vespa · Jina — organized by retrieval-pipeline layer. The default loop is offline and deterministic; every model-bearing or LLM technique is opt-in and leaves the default byte-identical (ADR 0001/0005).

The moat competitors don't have: enforced anti-fabrication + source provenance, the spine family's offline OCR & strong-table extraction (pdf→ppt→doc, pdfspine→ocrspine, scanned PDFs actually OCR'd on CPU), and an offline deterministic charter — the whole pipeline runs without a network, a GPU, or a framework.

Architecture

A deep, domain-grouped package layout — find the file by folder before you read a name.

src/ragspine/
├── common/         cross-cutting: company profile, sensitivity, glossary, observability
├── extraction/     documents → a frozen StyledGrid intermediate representation (IR)
│   ├── extractors/   xlsx / pptx / pdf (digital + scanned/OCR), style- & color-aware
│   ├── routing/      per-page PDF triage (digital vs scanned vs export)
│   ├── color/        controlled color-semantics registry
│   └── verification/ dual-channel cross-check → review queue
├── ingestion/      IR/text → stores
│   ├── structured/   fact ingestion + batch manifest ledger (idempotent)
│   ├── narrative/    document chunk ingestion + extraction
│   └── review/       human review-queue state machine (SME)
├── storage/        fact store (numeric) + chunk store (narrative), sqlite, full lineage
├── retrieval/      narrative RAG
│   ├── chunking/     paragraph-granular chunker + versioned chunk store
│   ├── lexical/      Okapi BM25 (CJK uni+bigram) + RRF fusion
│   ├── vector/       injectable embedding backends (default: none = pure BM25)
│   ├── rerank/       LLM listwise reranker (RRF-fallback)
│   └── link/         adapter wiring retrieval into the agent (strips RESTRICTED at exit)
├── agent/          intent parsing, clarification gateway, tool-use loop, llm provider
├── eval/           QA + extraction evaluation harnesses with baseline gates
└── service/        FastAPI app, RQ task queue, ingestion jobs, FAQ short-circuit cache

Request flow

question
  → intent parse (metric / entity / period / channel)
  → clarification gate ──(ambiguous)→ ask  ──(out-of-scope entity)→ refuse
  → FAQ short-circuit (service edge) ──(vetted hit)→ cached answer + provenance
  → route:
       structured → function-calling over the fact store → found / not_found / unrecognized
       narrative  → hybrid retrieve → listwise rerank → synthesize with citations
       composite  → run both, compare, merge
  → answer + sources   (anti-fabrication guard rewrites to "not found" if no fact)

Install

pip install rag-spine      # distribution name is hyphenated; the import is:  import ragspine

Optional extras:

Install optional backends on demand — the lean default (pip install rag-spine) runs offline on pure BM25 with zero heavy deps; the quality-depth backends are all opt-in.

From source

git clone https://github.com/VoldemortGin/ragspine.git && cd ragspine
uv venv .venv
VIRTUAL_ENV="$(pwd)/.venv" uv pip install -e ".[dev,service,vector]"

Quickstart

0. Fastest smoke test — the installed ragspine CLI, offline, no API key:

ragspine quickstart    # one FOUND answer (with provenance) + one honest "not found" — proves anti-fabrication in seconds

Use your own documents — one workspace, no repository scripts:

ragspine ingest ./documents --workspace .ragspine
ragspine ask --workspace .ragspine "What changed and why?"

The same high-level path is available from Python. The local default is offline and uses the existing guarded agent path; advanced stores/providers remain available through the lower-level APIs.

from ragspine import RAGSpine

with RAGSpine.local(".ragspine") as rag:
    report = rag.ingest("./documents")
    result = rag.ask("What changed and why?")
    print(report.summary)
    print(result.answer, result.sources)

Choose a coherent retrieval preset instead of wiring individual backends:

RAGSpine.local(".ragspine", profile="economy")   # BM25, smallest/offline
RAGSpine.local(".ragspine", profile="balanced")  # deterministic hybrid, offline
RAGSpine.local(".ragspine", profile="quality")   # ONNX + rerank; install model extras

Inspect or progressively expand configuration, then launch the workspace locally without Redis:

ragspine config init --profile balanced
ragspine config show --config ragspine.toml --effective
ragspine doctor --config ragspine.toml
ragspine serve --workspace .ragspine --open   # requires rag-spine[service]

1. End-to-end demo on synthetic data — offline, no API key:

.venv/bin/python scripts/run_demo.py        # → ALL CHECKS PASSED

2. Ask a question (offline deterministic MockProvider):

Needs the fact store from step 1 — scripts/run_demo.py populates the gitignored data/fact_metric.db. ragspine ask errors honestly if the db is missing; it never silently invents an empty one and returns a false "not found".

ragspine ask --db data/fact_metric.db "中国内地FY2024的REVENUE是多少"
# → ACME_CN FY2024 REVENUE 为 1320 USD_M（来源：ACME_FY2024_Review.pptx · slide=2,table=1,row=REVENUE,col=FY2024）

Ask for something the data doesn't have and you get an honest refusal, never a guess:

ragspine ask --db data/fact_metric.db "中国内地FY2025的REVENUE是多少"
# → 查不到：REVENUE / ACME_CN / 2025 …未在事实表中找到。为避免误导，不提供任何推测数字。

3. Python API:

from ragspine.agent.agent import answer_question
from ragspine.agent.llm_provider import MockProvider
from ragspine.storage.fact_store import SqliteFactStore

store = SqliteFactStore("data/fact_metric.db"); store.init_schema()
result = answer_question("中国内地FY2024的REVENUE是多少", store, MockProvider())
print(result.answer)     # deterministic value, or an honest "not found"
print(result.sources)    # [{'doc': ..., 'locator': ...}]

4. HTTP service + async ingestion:

# API
RAGSPINE_DB_PATH=data/fact_metric.db .venv/bin/python scripts/run_server.py --port 8000
curl -s localhost:8000/v1/ask -H 'content-type: application/json' \
     -d '{"question":"中国内地FY2024的REVENUE是多少"}'

# worker (needs Redis) — ingestion jobs run out-of-process
RAGSPINE_REDIS_URL=redis://localhost:6379/0 .venv/bin/python scripts/run_worker.py

Endpoints: GET /healthz, GET /readyz, POST /v1/ask, POST /v1/ingest/structured/jobs, POST /v1/ingest/narrative/jobs, GET /v1/jobs/{id}.

Core concepts

• Structured channel — every number lives in a fact_metric table with full lineage (source_doc_id + source_locator). A glossary normalizes ZH/EN/abbrev synonyms to controlled metric/entity/period codes (returns None rather than guessing). A function-calling query_metric tool returns found / not_found / unrecognized — and the agent never lets the model invent a number.
• Narrative channel — chunking → hybrid retrieval (BM25 + injectable vector + RRF k=60 + glossary multi-query) → optional LLM listwise_rerank → synthesis with citations. RESTRICTED-tier content is filtered at two exits before it can reach a prompt.
• Agent — four-slot intent parse → clarification gateway (answer-first, expose assumptions, one-click narrow) → route → anti-fabrication guard.
• Ingestion — extractors emit a frozen StyledGrid IR; structured & narrative ingestion are hash-idempotent; low-confidence/conflicting items go to a human review queue (distinct from the async job queue).
• FAQ cache — SME-vetted Q→A short-circuit with conservative exclusions: structured numeric questions, competitor/external entities, real-time queries, expired, disabled, and RESTRICTED items never short-circuit.
• Config — ServiceConfig (env-driven, RAGSPINE_*) + CompanyProfile (config/company.example.toml → copy to config/company.toml).

Extension points (just implement a Protocol)

LLMProvider · EmbeddingBackend · VectorStore · PersistencePolicy · ListwiseJudge · OcrBackend · Extractor · Chunker · NarrativeRetriever · NodePostprocessor · QueryDecomposer / query-transform · RelevanceGrader · EntailmentJudge · GraphStore · VisualEmbedder · TaskQueue — implement and inject. The core depends on the abstraction, never the SDK, so adding a provider / vector store / reranker / OCR engine touches one new file. The quality-depth seams are config-selectable by a make_* factory + RAGSPINE_* env, mirroring make_vector_store: reranker (cross-encoder / ColBERT / SPLADE via make_reranker), postprocessor (MMR / lost-in-the-middle / compression via make_postprocessor), query transform (HyDE / RAG-Fusion / step-back / adaptive via make_query_transform), chunker (layout / parent-child / sentence-window / semantic via make_chunker), make_graph_store, and make_visual_embedder — each with an offline default and its own provenance/isolation conformance pack. VectorStore ships a conformance kit (tests/conformance/) that binds provenance / isolation / determinism for any implementation; select one by config (make_vector_store / RAGSPINE_VECTOR_STORE), or let a third-party package register a backend by name via the ragspine.vector_stores entry-point group (no core PR). With a persistent store, NarrativeIndex embeds-and-persists at ingest (so a fresh process re-uses vectors, no re-embedding); the swappable PersistencePolicy gates what is written at rest — its default never persists a RESTRICTED chunk's vector.

Configuration

Testing

.venv/bin/python -m pytest tests/ -q -m "not gpu and not docling and not network"   # 2273 passed (the CI default)

The project is test-driven: tests are the spec. The gpu marker gates real-OCR / ColPali integration tests to a Linux + NVIDIA GPU box; the network marker gates the real-model (fastembed/HF "first-pull-then-offline") assertions — both are excluded by make ci. The pgvector conformance skips unless RAGSPINE_PG_URL points at a Postgres with the pgvector extension (the adapter is conformance-bound, just not in the default server-less CI). The Qdrant conformance runs unconditionally (local mode is purely in-process, no server). Everything else runs anywhere.

Continuous integration (local)

CI runs on your machine, not on GitHub Actions. scripts/ci.sh is the gate (full test suite, gpu-excluded, + demo smoke), and a pre-push hook enforces it so red code never gets pushed:

scripts/ci.sh                        # run the gate manually
git config core.hooksPath .githooks  # enable the pre-push gate (once per clone)

.github/workflows/ci.yml is included but dormant — manual-trigger only — so it consumes zero Actions minutes. Uncomment its push: / pull_request: triggers to enable server-side CI; it runs the exact same scripts/ci.sh. Lint / type-check (scripts/lint.sh, ruff + mypy) is opt-in and informational for now (the inherited code predates linting).

Demo data

The bundled demo uses a fictional company (ACME), synthetic figures, and a fictional competitor set — all generated by scripts/make_*.py (regenerable, deterministic). The version-controlled evaluation sets live under data/golden/. Nothing here is real-world data.

enterprise_pdf_rag（财务 PDF 可追溯 RAG 后端）

同仓库、同一个 pyproject.toml 里的 sibling package（import 名 enterprise_pdf_rag，不在 ragspine.* 命名空间下；决定与待办见 ADR 0021）。一句话定位：把财务 PDF 变成内容寻址、不可变的 source → processing → retrieval 快照，从 PDF span/drawing 到答案保留完整证据链，只让来源资格化的图表事实进入 ChartQA。它带来 CAS 快照、证据链与 chart QA；其 document-catalog 模式复用 RAGSpine 的 BM25 / RRF / listwise rerank 做 hybrid 检索，并在证据链上做一次模型调用 + 逐字段校验的回答（ADR 0011，离线验证；真实模型验收见其交接文档）。

• 安装：与 RAGSpine 共用一个 pyproject.toml —— uv sync（或上文 Install 的 uv pip install -e ".[dev,service,vector]"）即同时得到两个包与两个 console script。
• CLI：enterprise-pdf-rag ingest|qualify|index|publish|serve（另有 chart-qa / demo / extract / llm-smoke，以及 AIA 样本专用的 ingest-aia / process-aia-* / index-aia-processing）。
• 测试与门：bash scripts/ci.sh 已包含它的测试（第 5 步，与 ragspine 同一次 pytest 收集）和四个结构守卫（第 9 步：conformance / architecture / schema / drift）；单跑 uv run pytest tests/enterprise_pdf_rag -q。
• 文档入口：包内契约 src/enterprise_pdf_rag/CLAUDE.md 与规则 AGENTS.md；长文档在 docs/enterprise-pdf-rag/：README · 交接 · PRD v0.2 · ADR 0001–0011 · 测试与通用入库 · schemas。配置 config/enterprise-pdf-rag/settings.yaml，部署 deploy/enterprise-pdf-rag/，基准 data/benchmarks/enterprise-pdf-rag/。

Quickstart：财务 PDF 流水线

安装（上文 Install 的 uv pip install -e ".[dev,service,vector]" 或 uv sync）即得 enterprise-pdf-rag 命令；PDF 引擎 pdfspine 在基础依赖里，此流水线无需额外 extra。

前置 env（checkout 之外运行的约定见 ADR 0007）：APP_ROOT_DIR 指向已存在的工作目录、APP_DATA_DIR 指向持久数据目录；OPENAI_BASE_URL(https 强制)/OPENAI_MODEL/OPENAI_API_KEY 供 layout 阶段每页一次模型调用——--stage source 可零模型，但文档检索状态为 not_ready；EMBEDDING_BASE_URL 必须是回环地址（127.0.0.1/localhost/::1），远端模型走既有 SSH 隧道，另配 EMBEDDING_MODEL/EMBEDDING_API_KEY；RERANK_* 可选，仅请求显式 "rerank": true 时使用。

# 1) 入库：完整来源始终保存，选页只驱动下游；此命令的 JSON 输出给出后三步要用的 store 路径
enterprise-pdf-rag ingest --pdf report.pdf --pages all --stage semantics --max-live-calls N
# 取上一步输出的 source_store / processing_store / processing_id，记作 SRC / PROC / PID：
enterprise-pdf-rag qualify --source-store SRC --processing-store PROC --processing-id PID   # 资格化，零模型
enterprise-pdf-rag index   --source-store SRC --processing-store PROC --processing-id PID --document-label "FY25 report"   # 本地 embedding 一次
enterprise-pdf-rag publish --source-store SRC --processing-store PROC --processing-id PID   # 切指针并默认激活来源，零模型
# 起 document-catalog 服务并问答
APP_EXECUTION_MODE=document-catalog enterprise-pdf-rag serve --host 127.0.0.1 --port 8766
curl -s http://127.0.0.1:8766/v1/models   # 每个挂载文档一个 id：enterprise-pdf-rag/<sha12>
curl -s http://127.0.0.1:8766/v1/chat/completions -H 'Content-Type: application/json' \
  --data '{"model":"enterprise-pdf-rag/<sha12>","messages":[{"role":"user","content":"FY25 revenue?"}]}'

回答链保证：每条 claim 逐字段回读证据，散文里任何未验证的数字都触发整体拒答（HTTP 200 + enterprise_pdf_rag.status=abstained），从不派生数值。

Python API：同一流水线见 enterprise_pdf_rag.adapters.pdf_ingestion.ingest_pdf 与 adapters.draft_publication 的 qualify_draft / index_draft / publish_draft。完整命令、env 与契约见 测试与通用入库。

Status & roadmap

Current release: 0.11.0. It adds a 1,000-template workflow catalog, natural-language scaffolding, Dify/n8n interoperability, and versioned graph previews on top of the completed twelve-workstream quality-depth program (docs/prd-quality-depth.md):

• This release (0.11.0): bounded offline catalog loading and semantic matching, JSON/YAML/TOML workflow input, Dify-compatible scaffolding, n8n conversion, public graph-preview generation, CLI/API access, and the website export contract for all 1,000 bundled templates.
• 0.10.0: metadata filtering, multi-library routing, an economy retrieval preset, store-level parent-child expansion, and conversation history that only enters generation context.
• 0.8.0 (W8–W12, the second competitor batch): post-retrieval postprocessor chain (MMR de-dup · lost-in-the-middle · context compression), LLM query transforms (HyDE · RAG-Fusion · step-back · Adaptive-RAG), RAPTOR recursive-summary tree + sentence-window / semantic chunking, ColBERT late-interaction + SPLADE learned-sparse rerankers, and ColPali page-as-image visual retrieval.
• 0.7.0 (W1–W7, the first batch): true semantic ONNX embedding default + dense-on hybrid, local cross-encoder rerank, family OCR / .docx / .pptx / rich-table extraction, contextual + parent-child chunking, the faithfulness/groundedness eval gate, agentic decomposition / CRAG / self-RAG / multi-turn, and the deterministic GraphRAG relation graph + GraphStore seam.

Solid: structured channel, narrative hybrid retrieval, agent orchestration, office extraction (xlsx/pptx/pdf), FastAPI + RQ service, FAQ cache, evaluation harness, 2273 tests.

Honest follow-ups (contributions welcome): the depth workstreams ship the technique + conformance; their measured eval-delta (real-model A/B ratchets on the W5 groundedness harness) is the follow-up — a depth item isn't "done" until the ratchet shows it improved the answer. ColBERT / SPLADE / ColPali land as rerankers / a visual seam; the heavy multi-vector & sparse retrieval indexes (and ColPali's real GPU end-to-end) are follow-ups. The groundedness gate defaults to offline lexical-overlap entailment; the real ONNX-NLI / RAGAS judge is opt-in behind the seam. The VectorStore seam is wired live with three real adapters — HybridRetriever delegates vector scoring to it (byte-identically), it's config-selectable (make_vector_store / RAGSPINE_VECTOR_STORE), named in .topology(), and behind [vector] ships sqlite-vec (embedded) + pgvector (Postgres, BSD pg8000 driver) + qdrant (HNSW, local mode, Apache-2.0 qdrant-client) — all persistent and conformance-bound; still open are more adapters (Milvus/FAISS) and true ANN (the adapters persist but currently score exactly in Python, not via native HNSW/IVFFlat KNN). With [embed-onnx] the default is genuinely semantic (BM25 + ONNX MiniLM → RRF, CPU-only); with no extra it falls back to pure BM25. Pipeline-topology export (.topology() → Mermaid/DOT/JSON, plus scripts/topology.py) ships — see src/ragspine/pipeline/.

License

Apache License 2.0. See NOTICE.