OpenQuery

Query public data sources worldwide through a unified CLI and REST API.

OpenQuery provides a plugin-based framework for scraping government websites, public registries, and open data APIs. It handles the hard parts — browser automation, CAPTCHA solving, WAF bypass, caching, and rate limiting — so you can focus on the data.

Current rollout note: wave 1 is America-first. The machine-readable America inventory lives in docs/americas-source-inventory.json, and docs/sources.md derives public inventory counts from that snapshot. Callable INTL runtime connectors remain outside the America completeness contract for this wave, and the repo intentionally defers a new typecheck gate until a later wave.

Features

• Unified interface — one CLI and one API endpoint for all data sources
• Browser automation — Playwright-based scraping for JavaScript-heavy sites
• Multi-engine CAPTCHA solving — PaddleOCR (100%), EasyOCR+Tesseract voting (90%), with cloud and paid fallbacks
• LLM-powered knowledge CAPTCHAs — Ollama (local), HuggingFace, Anthropic, OpenAI fallback chain
• Audit & evidence — screenshots, network logs, and PDF evidence reports for compliance
• WAF bypass — browser-context API calls preserve session cookies
• Caching — in-memory, Redis, or SQLite backends with configurable TTL
• Rate limiting — per-source token-bucket to respect server limits
• REST API — FastAPI server with auto-generated OpenAPI docs
• Document OCR — extract structured data from ID documents (cedula, INE, DNI, carnet, passport)
• Face verification — 1:1 face comparison with liveness detection (DeepFace/ArcFace)
• Health monitoring — per-source circuit breaker with automatic failover
• Dashboard — web UI for source browsing, querying, and health monitoring
• Extensible — add new data sources by implementing a single class
• Country-organized — sources grouped by country code (co, us, etc.)

Built-in Sources — 562 sources across 21 country and region namespaces

For current America rollout status, use docs/sources.md and docs/americas-source-inventory.json as the source of truth. Public inventory counts there are derived from the America snapshot; docs/test_results.md is a live accountability report and can include callable runtime connectors that are outside the America inventory contract (for example, INTL).

Colombia

Antecedentes y Justicia

Identidad y Registro Civil

Compliance y AML

Seguridad Social

Empresas y Comercio

Propiedad e Inmuebles

Vehículos y Tránsito

Vivienda y Servicios

Turismo

Salud

Consejos Profesionales

United States

Ecuador

Peru

Chile

Mexico

Argentina

Brazil (1 source)

Costa Rica (1 source)

Dominican Republic (1 source)

Paraguay (1 source)

Guatemala (1 source)

Honduras (1 source)

International

Installation

pip install openquery

Or with uv:

uv add openquery

System Dependencies

Playwright browsers are required for web scraping:

playwright install chromium

CAPTCHA Engines (pick one or more)

OpenQuery auto-detects installed OCR engines and builds an optimal solver chain:

For knowledge-based CAPTCHAs (Procuraduria), you need at least one LLM backend:

Optional Extras

pip install "openquery[paddleocr]"   # PaddleOCR — best CAPTCHA accuracy (100%)
pip install "openquery[easyocr]"     # EasyOCR — good accuracy (85%), combines with Tesseract for 90%
pip install "openquery[huggingface]" # HuggingFace Inference API (OCR + QA)
pip install "openquery[serve]"       # FastAPI server + dashboard (fastapi, uvicorn)
pip install "openquery[redis]"       # Redis cache backend
pip install "openquery[captcha]"     # 2captcha paid CAPTCHA solving (last resort)
pip install "openquery[deepface]"    # Face verification (DeepFace + ArcFace)
pip install "openquery[passport]"    # Passport MRZ reading (passporteye)

Quick Start

CLI

# List available data sources
openquery sources

# Query Colombian traffic fines by cedula
openquery query co.simit --cedula 12345678

# Query Colombian vehicle registry by plate
openquery query co.runt --placa ABC123

# Query by VIN
openquery query co.runt --vin 5YJ3E1EA1PF000001

# Disciplinary records
openquery query co.procuraduria --cedula 12345678

# Criminal background
openquery query co.policia --cedula 12345678

# Health system enrollment
openquery query co.adres --cedula 12345678

# Pico y placa — is my plate restricted today?
openquery query co.pico_y_placa --placa ABC123

# Toll tariffs
openquery query co.peajes --custom peaje --extra '{"peaje": "ALVARADO"}'

# Fuel prices in Bogota
openquery query co.combustible --custom fuel --extra '{"municipio": "BOGOTA"}'

# EV charging stations in Medellin
openquery query co.estaciones_ev --custom ev --extra '{"ciudad": "Medellin"}'

# Road crash hotspots
openquery query co.siniestralidad --custom stats --extra '{"departamento": "CUNDINAMARCA"}'

# Vehicle fleet lookup by plate
openquery query co.vehiculos --placa ABC123

# Output raw JSON
openquery query co.simit --cedula 12345678 --json

# Generate audit evidence (screenshots + PDF report)
openquery query co.runt --placa ABC123 --audit --audit-dir ./evidence

# Source health status
openquery health

# Extract data from ID document photo
openquery ocr --type co.cedula cedula_photo.jpg

# Face verification (compare ID photo vs selfie)
openquery face-verify id_photo.jpg selfie.jpg

REST API

# Start the API server
openquery serve

# Or with custom host/port
openquery serve --host 127.0.0.1 --port 3000

Then query via HTTP:

curl -X POST http://localhost:8000/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "source": "co.simit",
    "document_type": "cedula",
    "document_number": "12345678"
  }'

Response:

{
  "ok": true,
  "source": "co.simit",
  "queried_at": "2026-03-31T10:30:00Z",
  "cached": false,
  "latency_ms": 4523,
  "data": {
    "comparendos": 0,
    "multas": 0,
    "total_deuda": 0.0,
    "paz_y_salvo": true
  }
}

API Endpoints:

Docker

docker compose up

This starts the API server with Redis caching on port 8000.

Configuration

All settings use environment variables with the OPENQUERY_ prefix:

Adding a New Source

Create a new source by implementing the BaseSource class:

# src/openquery/sources/us/nhtsa.py
from pydantic import BaseModel
from openquery.sources import register
from openquery.sources.base import BaseSource, DocumentType, QueryInput, SourceMeta


class NhtsaResult(BaseModel):
    manufacturer: str = ""
    model: str = ""
    year: int = 0
    recalls: list[dict] = []


@register
class NhtsaSource(BaseSource):
    def meta(self) -> SourceMeta:
        return SourceMeta(
            name="us.nhtsa",
            display_name="NHTSA Vehicle Safety",
            description="US vehicle safety recalls and VIN decoding",
            country="US",
            url="https://vpic.nhtsa.dot.gov/api/",
            supported_inputs=[DocumentType.VIN],
            requires_captcha=False,
            requires_browser=False,
            rate_limit_rpm=30,
        )

    def query(self, input: QueryInput) -> NhtsaResult:
        import httpx
        resp = httpx.get(
            f"https://vpic.nhtsa.dot.gov/api/vehicles/decodevin/{input.document_number}",
            params={"format": "json"},
        )
        data = resp.json()
        # Parse and return NhtsaResult...

The @register decorator automatically makes the source available in the CLI, API, and source listing.

Architecture

openquery/
├── core/
│   ├── browser.py    # Playwright browser management
│   ├── captcha.py    # Multi-engine CAPTCHA solvers (PaddleOCR, EasyOCR, Tesseract, voting)
│   ├── llm.py        # LLM QA chain (Ollama, HuggingFace, Anthropic, OpenAI)
│   ├── audit.py      # Evidence capture (screenshots, network logs, PDF reports)
│   ├── cache.py      # Caching backends (memory, Redis, SQLite)
│   └── rate_limit.py # Token-bucket rate limiting
├── sources/          # Data source plugins, organized by country
│   ├── base.py       # BaseSource ABC — implement this to add sources
│   ├── co/           # Colombia
│   ├── ec/           # Ecuador
│   ├── pe/           # Peru
│   ├── cl/           # Chile
│   ├── mx/           # Mexico
│   ├── ar/           # Argentina
│   ├── us/           # United States
│   └── intl/         # International
├── models/           # Pydantic response models, organized by country
├── server/           # FastAPI REST API
└── commands/         # Typer CLI commands

Development

git clone https://github.com/dacrypt/openquery.git
cd openquery
uv sync --all-extras
playwright install chromium

# Run tests
uv run pytest

# Lint
uv run ruff check src/ tests/

See CONTRIBUTING.md for detailed guidelines.

Documentation

License

MIT