Multi-engine document OCR with cascading fallback
Project description
smart-ocr
Multi-engine OCR with cascading fallback, quality audit, and figure extraction.
Process academic papers and documents using free local models first, with automatic cloud fallback for failed pages. Extract and describe figures using vision models.
Features
- Multi-engine OCR — Nougat, DeepSeek, Mistral, Gemini (via dedicated CLI tools)
- Smart routing — Free local engines first, cloud only when needed
- Quality audit — Heuristics + LLM review catches garbage text
- Figure extraction — Renders figures from PDFs, describes with vision models
- Parallel processing — Process multiple pages/figures concurrently
- Batch processing — Process entire directories of papers
- CLI — Live progress, colored panels, cost tracking
- Modular engines — Each OCR backend is a standalone CLI tool
Quick Start
# Install globally (recommended)
pipx install smart-ocr --python python3.12
# Or install in a project
pip install smart-ocr
# Pull required Ollama models
ollama pull deepseek-r1:32b # quality audit
ollama pull deepseek-ocr # local OCR
# Process a paper
smart-ocr paper.pdf
# Process with figure images saved
smart-ocr paper.pdf --save-figures
# Batch process a folder
smart-ocr batch ~/Papers/ --limit 10
Example Output
Processing a 22-page economics paper:
smart-ocr v0.1.0
kuttner_2001_monetary_policy.pdf
22 pages, 1.2 MB
type: academic
(1) primary ocr
deepseek
[+] page 1
[+] page 2
...
(2) quality audit
[!] page 10 (19.2% garbage)
(3) fallback ocr
gemini
[+] page 10
(4) figure processing
[+] fig 1 (p.1): unknown
[+] fig 2 (p.8): scatter_plot
[+] fig 3 (p.12): scatter_plot
---
done 22/22 pages
3 figures
241.5s
$0.0002
deepseek (21) + gemini (1)
-> output/kuttner_2001_monetary_policy/kuttner_2001_monetary_policy.md
Output structure:
output/<doc_stem>/
├── <doc_stem>.md # Full OCR text + figure descriptions
├── metadata.json # Stats, engines used, cost
└── figures/ # With --save-figures
├── figure_1_page1.png
└── ...
Output Formats
Use -f/--format to choose the output format:
| Format | Content | Best For |
|---|---|---|
markdown (default) |
Formatted text with page headers, inline figure descriptions | Human reading, documentation |
json |
Structured data with pages, figures, stats, engine info | Programmatic access, downstream processing |
txt |
Plain concatenated text with page separators | Simple text extraction |
JSON format structure
{
"document": "/path/to/paper.pdf",
"pages": [
{
"page_num": 1,
"text": "...",
"status": "success",
"engine": "deepseek",
"confidence": null,
"figures": [
{
"figure_num": 1,
"figure_type": "scatter_plot",
"description": "A scatter plot showing...",
"bbox": [100, 200, 400, 500],
"image_path": "figures/figure_1_page1.png"
}
]
}
],
"stats": {
"total_pages": 22,
"pages_success": 22,
"figures_detected": 3,
"total_cost": 0.0002,
"total_time": 241.5
}
}
The metadata.json contains:
{
"document": "/path/to/paper.pdf",
"stats": {
"total_pages": 22,
"pages_success": 22,
"total_cost": 0.0002,
"total_time": 241.5
},
"engines_used": { "deepseek": 21, "gemini": 1 },
"figures": 3,
"pages_needing_reprocessing": []
}
See examples/ for complete processed papers:
kuttner_2001/- 22 pages, 3 figures (monetary policy)sutskever_2014/- 9 pages (sequence to sequence learning)bernanke_kuttner_2005/- 37 pages, 6 vector figures (stock market reaction)
Pipeline Architecture
┌─────────────┐
│ PDF Input │
└──────┬──────┘
│
▼
┌────────────────────────┐
│ Document Classifier │
│ (academic vs general) │
└───────────┬────────────┘
│
┌───────────────────────────────┼───────────────────────────────┐
│ │ │
▼ ▼ ▼
ACADEMIC PATH GENERAL PATH ROUTER AGENT
(Nougat → DeepSeek) (DeepSeek → Nougat) (picks available engine)
│ │ │
└───────────────────────────────┴───────────────────────────────┘
│
┌───────────────┴───────────────┐
│ STAGE 1: PRIMARY OCR │
│ │
│ ┌─────────────────────────┐ │
│ │ Local Engines (FREE): │ │
│ │ │ │
│ │ • Nougat │ │
│ │ nougat_ocr lib │ │
│ │ model: 0.1.0-small │ │
│ │ │ │
│ │ • DeepSeek │ │
│ │ Ollama │ │
│ │ deepseek-ocr:latest │ │
│ └─────────────────────────┘ │
│ OR │
│ ┌─────────────────────────┐ │
│ │ Cloud Engines (PAID): │ │
│ │ │ │
│ │ • Gemini │ │
│ │ Google genai SDK │ │
│ │ gemini-3-flash │ │
│ │ ~$0.0002/page │ │
│ │ │ │
│ │ • Mistral │ │
│ │ Mistral SDK │ │
│ │ pixtral-large │ │
│ │ ~$0.001/page │ │
│ └─────────────────────────┘ │
└───────────────┬───────────────┘
│
[Page Results]
│
▼
┌───────────────────────────────┐
│ STAGE 2: QUALITY AUDIT │
│ │
│ ┌─────────────────────────┐ │
│ │ Heuristics Checker │ │
│ │ (rule-based, instant): │ │
│ │ • Word count ≥ 50 │ │
│ │ • Garbage ratio < 15% │ │
│ │ • Avg word length ok │ │
│ │ • No unicode issues │ │
│ │ • No repeated patterns │ │
│ └──────────┬──────────────┘ │
│ │ │
│ [pages flagged?] │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ │
│ │ Cross-Check (optional): │ │
│ │ Try 2nd local engine │ │
│ └──────────┬──────────────┘ │
│ │ │
│ [still flagged?] │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ │
│ │ LLM Auditor (FREE): │ │
│ │ Ollama │ │
│ │ deepseek-r1:32b │ │
│ │ (reasoning model) │ │
│ │ • Can override heur. │ │
│ │ • verdict: acceptable, │ │
│ │ needs_review, poor │ │
│ └──────────┬──────────────┘ │
└─────────────┼─────────────────┘
│
┌──────────────┴──────────────┐
│ │
[all pass] [some flagged]
│ │
│ ▼
│ ┌───────────────────────────────┐
│ │ STAGE 3: FALLBACK OCR │
│ │ │
│ │ Router selects different │
│ │ engine (not primary): │
│ │ Gemini → Mistral → │
│ │ DeepSeek → Nougat │
│ │ │
│ │ Reprocess flagged pages │
│ └───────────────┬───────────────┘
│ │
└─────────────┬───────────────┘
│
[All Pages]
│
▼
┌────────────────────────────┐
│ STAGE 4: FIGURE AGENT │
│ (if enabled) │
│ │
│ ┌──────────────────────┐ │
│ │ PyMuPDF Extractor: │ │
│ │ • Vector figures │ │
│ │ (charts, plots) │ │
│ │ • IMAGE blocks │ │
│ │ • Embedded images │ │
│ │ • Filter by size │ │
│ │ • Render at 150 DPI │ │
│ └──────────┬───────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Vision Engine: │ │
│ │ Gemini 3 Flash │ │
│ │ DeepSeek-OCR │ │
│ │ Pixtral-Large │ │
│ └──────────┬───────────┘ │
│ │ │
│ [Figure Results] │
└─────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ OUTPUT ASSEMBLY │
│ │
│ Format: markdown/json/txt │
│ │
│ output/<doc_stem>/ │
│ ├─ document.md │
│ ├─ metadata.json │
│ └─ figures/ │
│ └─ figure_N.png │
└─────────────────────────────┘
Cost optimization:
- Local engines first (Nougat, DeepSeek via Ollama) - FREE
- Quality audit with local Ollama (llama3.2/qwen2.5) - FREE
- Cloud fallback only for failed pages (Gemini ~$0.0002/page)
- Typical 22-page paper: $0.0002 total (only 1 page needed cloud)
Requirements
Base:
- Python 3.11+
- Ollama with
deepseek-r1:32b(for quality audit) orllama3.2/qwen2.5as lighter alternatives
OCR Engines (install individually based on needs):
| Engine | Installation | Type | Cost |
|---|---|---|---|
| DeepSeek | pip install deepseek-ocr-cli |
Local (Ollama) | Free |
| Nougat | pip install nougat-ocr-cli |
Local (Python) | Free |
| Gemini | pip install gemini-ocr-cli |
Cloud API | ~$0.0002/page |
| Mistral | pip install mistral-ocr-cli |
Cloud API | ~$0.001/page |
Installation examples:
# Install with local engines only (recommended to start)
pip install smart-ocr[local]
# Install with all engines
pip install smart-ocr[all]
# Install specific engines
pip install smart-ocr[deepseek,gemini]
Setup:
# 1. Install Ollama and pull audit model (reasoning model recommended)
ollama pull deepseek-r1:32b # best quality, or llama3.2 for speed
# 2. If using DeepSeek OCR, pull its model
ollama pull deepseek-ocr:latest
# 3. If using cloud engines, set API keys
export GEMINI_API_KEY="your-key"
export MISTRAL_API_KEY="your-key"
# 4. Check engine status
smart-ocr engines
OCR Engine CLIs
Each OCR backend is a standalone CLI tool that can be used independently:
- deepseek-ocr-cli - Local OCR via Ollama/DeepSeek
- gemini-ocr-cli - Cloud OCR via Google Gemini
- mistral-ocr-cli - Cloud OCR via Mistral AI
- nougat-ocr - Academic papers (used as Python library)
CLI Commands
# Simple usage
smart-ocr paper.pdf # Process a PDF
smart-ocr paper.pdf --save-figures # Save figure images
# Full options
smart-ocr process paper.pdf [OPTIONS]
-o, --output PATH Output file path
-f, --format markdown|json|txt
--primary ENGINE Force primary engine
--fallback ENGINE Force fallback engine
--no-audit Skip quality audit
--no-figures Skip figure processing
--save-figures Save figure images to disk
--figures-engine gemini|deepseek|mistral (default: auto-select)
--timeout SECONDS Timeout per page/figure (default: 300)
--workers N Parallel workers (default: 4, use 1 for sequential)
# Batch process directory
smart-ocr batch ~/Papers/ [OPTIONS]
--limit N Process first N files
--save-figures Save all figure images
--figures-engine gemini|deepseek|mistral (default: auto-select)
--timeout SECONDS Timeout per page/figure (default: 300)
--workers N Parallel workers (default: 4)
# Check engines
smart-ocr engines
# Check audit system
smart-ocr audit-status
Parallel Processing
smart-ocr can process multiple pages and figures in parallel to speed up large documents:
# Fast processing with 8 parallel workers (recommended for M3 Pro/Max with 32GB+ RAM)
smart-ocr paper.pdf --workers 8
# Conservative processing for limited RAM
smart-ocr paper.pdf --workers 2
# Sequential processing (most reliable, slowest)
smart-ocr paper.pdf --workers 1
# Extended timeout for very complex pages (10 min per page)
smart-ocr paper.pdf --timeout 600
Hardware recommendations:
| RAM | Workers | Notes |
|---|---|---|
| 8GB | 1-2 | Sequential or minimal parallelism |
| 16GB | 2-4 | Good for most documents |
| 32GB | 4-6 | Fast processing |
| 64GB+ | 6-8 | Maximum parallelism |
Parallel processing uses ThreadPoolExecutor for concurrent page/figure processing. Each worker loads the page image into memory, so higher worker counts require more RAM.
Timeout configuration:
The default 300s (5 min) timeout works for most pages. For documents with complex figures or slow API responses, increase the timeout:
# smart-ocr.yaml
parallel_pages: 4
parallel_figures: 2
figure_timeout: 600 # 10 min per figure
nougat:
timeout: 600
deepseek:
timeout: 600
Configuration
Create smart-ocr.yaml in your project or home directory:
# Engine selection
primary_engine: deepseek
fallback_engine: gemini
figures_engine: gemini # For figure descriptions
# Quality audit
audit:
enabled: true
min_word_count: 50
garbage_threshold: 0.15
# Figure processing
include_figures: true
save_figures: false
figures_max_total: 25
figures_max_per_page: 3
figure_timeout: 180 # seconds per figure description
# Parallel processing (adjust based on your hardware)
parallel_pages: 4 # pages processed concurrently
parallel_figures: 2 # figures described concurrently
# Output
output_dir: output
output_format: markdown
# Engine-specific timeouts (seconds)
nougat:
timeout: 300
deepseek:
timeout: 300
gemini:
timeout: 300
mistral:
timeout: 300
Architecture
See docs/ARCHITECTURE.md for detailed module documentation.
src/smart_ocr/
├── cli.py # Click CLI
├── core/
│ ├── config.py # AgentConfig dataclass
│ ├── document.py # PDF loading
│ └── result.py # OCRResult, PageResult, FigureResult
├── engines/
│ ├── base.py # BaseEngine ABC
│ ├── deepseek.py # Ollama/DeepSeek
│ ├── gemini.py # Google Gemini
│ ├── mistral.py # Mistral AI
│ └── nougat.py # Nougat (academic)
├── audit/
│ ├── heuristics.py # Garbage detection
│ └── llm_audit.py # Ollama-based review
├── pipeline/
│ ├── processor.py # 4-stage pipeline
│ └── router.py # Engine selection
└── ui/ # Rich console output
License
MIT
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
smart_ocr-0.2.2.tar.gz
(567.9 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
smart_ocr-0.2.2-py3-none-any.whl
(74.3 kB
view details)
File details
Details for the file smart_ocr-0.2.2.tar.gz.
File metadata
- Download URL: smart_ocr-0.2.2.tar.gz
- Upload date:
- Size: 567.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4df30f269fde84dd081ca5c82599af2f0195310b8e1ae9bb1b9f5a4371de35b3
|
|
| MD5 |
940652a913c4f6c036110159b5898c09
|
|
| BLAKE2b-256 |
dba63e654f2fdec8e741ea8df50f5ac18360b11aa07b40878e40c5b51f0cf977
|
File details
Details for the file smart_ocr-0.2.2-py3-none-any.whl.
File metadata
- Download URL: smart_ocr-0.2.2-py3-none-any.whl
- Upload date:
- Size: 74.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
eda56d63faa652e718b971df9a14a16996d0c4d31c1502557693b93367131f78
|
|
| MD5 |
43b11f4b8e1958556bbfc942e965cc5a
|
|
| BLAKE2b-256 |
bde41822955b3938bb2187723356b4070baf928fcfcb033bdfbae09c52380c3a
|
