deepresearch-flow 0.16.0

ai-deepresearch-flow

From documents to deep research insight — automatically.

English | 中文

Core Pain Points

• OCR Chaos: Raw markdown from OCR tools is often broken — tables drift, formulas break, references are non-clickable.
• Translation Nightmares: Translating technical papers often destroys code blocks, LaTeX formulas, and table structures.
• Information Overload: Extracting structured insights (authors, venues, summaries) from hundreds of PDFs manually is impossible.
• Context Switching: Managing PDFs, summaries, and translations in different windows kills focus.

Solution

DeepResearch Flow provides a unified pipeline to Repair, Translate, Extract, and Serve your research library.

Key Features

• Smart Extraction — Turn unstructured Markdown into schema-enforced JSON (summaries, metadata, Q&A) using LLMs.
• Precision Translation — Translate OCR Markdown to Chinese/Japanese while freezing formulas, code, tables, and references.
• Local Knowledge DB — Web UI with Split View (Source/Translation/Summary), full-text search, and multi-dimensional filtering.
• Snapshot + API Serve — Production-ready SQLite snapshot with static assets and read-only JSON API.
• OCR Post-Processing — Fix broken references, merge split paragraphs, repair LaTeX and Mermaid diagrams.
• Semantic Search — LanceDB-backed vector search with hybrid recall and cloud reranking.
• MCP Integration — FastMCP server for AI agent access with bounded read tools, static-bearer Streamable HTTP/SSE, and GitHub OAuth at /oauth/mcp.

Quick Start

1) Installation

uv pip install deepresearch-flow
# or: pip install deepresearch-flow

2) Configuration

cp config.example.toml config.toml

Minimal config with weighted multi-provider routing:

main_model = [
  { model = "openai/gpt-4o-mini", weight = 4 },
  { model = "claude/claude-sonnet-4-5-20250929", weight = 1 }
]

[[providers]]
name = "openai"
type = "openai_compatible"
base = [
  { url = "https://api.openai.com/v1", weight = 1, key = [
    { value = "env:OPENAI_API_KEY", weight = 4 }
  ] }
]
models = [
  { model_name = "gpt-4o-mini", is_support_json_schema = true }
]

[[providers]]
name = "claude"
type = "claude"
base = [
  { url = "https://api.anthropic.com", weight = 1, key = [
    { value = "env:ANTHROPIC_API_KEY", weight = 1 }
  ] }
]
models = [
  { model_name = "claude-sonnet-4-5-20250929" }
]

Keys use env:VAR_NAME syntax to keep secrets out of config files. Multiple providers (Ollama, Gemini, DashScope, Azure OpenAI) are supported. For full configuration options (embedding, rerank, translator defaults, search), see config.example.toml.

3) Process a batch of PDFs

Prepare a project directory that holds pdf/ and exactly one .bib file, then generate and run run.sh:

mkdir myproj
cp -r /path/to/pdfs myproj/pdf
cp /path/to/library.bib myproj/

cp ocr.example.toml ocr.toml     # fill in the OCR backend token
uv run deepresearch-flow utils init-run ./myproj

Open myproj/run.sh and replace each --model PROVIDER/MODEL with a provider/model declared in config.toml. Different steps can use different models. Then run:

./myproj/run.sh

The script runs OCR, formula and format repair, structured summary extraction, and translation, and finally builds the snapshot database in myproj/dist/. Failed items are retried automatically, and a rerun after an interruption resumes at the failed step. For details, see From PDFs to a snapshot.

4) Browse the result

After run.sh finishes, open the local web UI:

uv run deepresearch-flow paper db serve \
  --input myproj/deep_read.json \
  --bibtex myproj/library.bib \
  --md-root myproj/md_simple \
  --md-translated-root myproj/md_base64_translated \
  --pdf-root myproj/pdf

To deploy the snapshot and static assets in production, see the Deployment Guide.

Documentation

Built with love for the Open Science community.