🧐 Skeptic — a self-fact-checking research agent

Most AI researchers confidently make things up. Skeptic doesn't — because a dedicated agent's only job is to disprove the others. Every claim ships with a citation and a confidence tag, or it gets flagged and sent back for better sources.

100% local and free. Runs entirely on your machine — no API keys, no per-token cost, works offline once the models are pulled.

Built with LangGraph · Ollama (local Llama 3.2 / Qwen 2.5) · MCP (DuckDuckGo search) · local RAG · Typer + Rich CLI.

How it works

   START
     │
     ▼
 ┌─────────┐
 │ Planner │  splits the question into focused sub-questions
 └─────────┘
     │
     ▼
 ┌──────────────┐   (parallel)   MCP web search + your local docs ─► RAG re-ranking
 │ Researchers  │  each sub-question → claims + real source URLs
 └──────────────┘
     │
     ▼
 ┌─────────┐   scores every claim: verified / weak / unverifiable
 │ Skeptic │──┐
 └─────────┘  │  too many weak claims + retries left?
     │        └──────────────► back to Researchers
     ▼ (good enough)
 ┌────────────┐
 │ Human gate │  you approve / reject / revise before it finalizes
 └────────────┘
     │
     ▼
 ┌─────────────┐
 │ Synthesizer │  writes the report, tags each claim ✅ / ⚠️ / ❌
 └─────────────┘
     │
     ▼
    END

The loop back to the researchers is the whole idea: weak evidence doesn't get published, it gets re-investigated. Search is an MCP tool, and each set of hits is RAG-ranked locally so the model reads the passages that matter.

Model routing (all local via Ollama)

Every model is overridable via env var (see .env.example) — bump to a bigger model for better answers, or drop to a smaller one for less RAM.

Quickstart

First install Ollama and start it:

ollama serve        # leave running in another terminal

Then install Skeptic as a global command (needs Python 3.10+):

pipx install .      # or: pip install .

Bootstrap the local models (one-time download), then ask away — from any folder:

skeptic setup                                        # pulls the models
skeptic ask "How does caffeine affect sleep quality?"

No keys, ever. The run prints a live agent-by-agent trace, pauses at a human gate for your approval, and saves the final report to ./reports/<timestamp>.md.

Handy commands:

skeptic doctor      # check Ollama + models are ready
skeptic models      # show which models each role uses
skeptic docs        # list the local documents indexed for retrieval

Prefer not to install? From this folder you can also run python cli.py ask "your question" inside the venv.

Fact-check against your own documents

Skeptic can ground answers in your own files — notes, papers, reports — right alongside the web. Drop .pdf, .txt, or .md files into ~/.skeptic/docs (or any folder) and they're chunked, embedded locally, and retrieved per sub-question just like web hits. Still 100% local; nothing leaves your machine.

mkdir -p ~/.skeptic/docs
cp ~/Downloads/*.pdf ~/.skeptic/docs/

skeptic docs                       # see what's indexed
skeptic docs --reindex             # embed them now (otherwise done on next ask)
skeptic ask "what does my research say about X?"

# or point one run at a specific folder:
skeptic ask --docs ./my-papers "summarize the key findings"

Citations to local files appear as file://… URLs. Embeddings are cached, so re-runs are fast and only re-embed when the files change. Override the default folder anytime with SKEPTIC_DOCS_DIR.

Project layout

skeptic/
├── state.py            # the shared graph state
├── models.py           # typed contracts every agent speaks in
├── config.py           # model routing + Ollama LLM factory
├── prompts.py          # all prompts in one place
├── tools.py            # web search as an MCP client (async)
├── mcp_server.py       # the DuckDuckGo MCP search server
├── rag.py              # local embed + cosine re-ranking of web results
├── docs.py             # local-document RAG (PDF/txt/md) + on-disk cache
├── memory.py           # SQLite checkpointer (durable/resumable runs)
├── graph.py            # wires the agents + the retry loop + human gate
├── cli.py              # the Typer + Rich `skeptic` command
└── agents/
    ├── planner.py
    ├── researcher.py
    ├── skeptic.py      # the unique hook
    ├── human_gate.py   # pause for human approval
    └── synthesizer.py
cli.py                  # backward-compat shim (python cli.py …)
pyproject.toml          # packaging → the global `skeptic` command

Roadmap

• Phase 1 — Planner → Researchers → Synthesizer
• Phase 2 — Skeptic + the falsify/retry loop
• Phase 3 — human approval gate + cross-run memory
• Phase 4 — MCP-native search (DuckDuckGo), local RAG, Ollama, global CLI
• Phase 5 — RAG over your own PDF/txt/md docs · publish-ready packaging

See HANDOFF.md for exact status, and PUBLISHING.md for how to ship it to PyPI.