DataSentry

Evidence-driven, local-first AI copilot for data quality.
Detect · Explain · Validate · Repair — with statistical evidence, AI assistance, and human approval.

中文导读：DataSentry 是一个以统计证据为基础、以 AI 为辅助、以人工审批为保障的本地优先数据质量平台。 一次扫描生成六维质量评分，每个问题带证据链；自然语言即可提出规则与修复方案，但只有人工批准才生效。 数据不出机器（LLM 可接本地 Ollama），DuckDB 执行引擎，百万行 10 秒级。

What is DataSentry?

DataSentry scans your data (CSV / Parquet / JSONL / XLSX / DuckDB / SQLite / PostgreSQL) and produces:

• 39 evidence-driven detectors — missingness, dates, encodings, cross-field rules, cross-table foreign keys, duplicates (exact + fuzzy), outlier models (Isolation Forest / LOF), and more. Every issue carries a statistical evidence chain: samples, ratios, confidence.
• Six-dimension quality score — completeness, validity, uniqueness, consistency, integrity, timeliness — with explainable weights and per-dimension contributions.
• Repair loop with human approval — propose → preview (rule re-run before/after) → apply (fingerprinted copy + rollback artifact) → rollback. AI suggests; you decide.
• Drift engine — compare historical scans: schema, row-count, score and issue-distribution drift.
• Quality gates in CI — scan --fail-on blocks releases by severity or score; export reports as JSON / Markdown / HTML / JUnit / SARIF.
• LLM assistance, safely — natural language → rule candidates with preflight simulation; PII redacted before any prompt; every call audited (llm status).
• Multiple surfaces — CLI, REST API, server-rendered Web UI with cross-scan trends, and an MCP stdio server so LLM agents can use the tools directly.

Live demo report — orders-report.html (200 rows with 15 injected quality issues)

Quick start

pip install datasentry-ai     # or: uv sync (source checkout)

datasentry scan orders.csv               # detect → fuse → score → persist, one step
datasentry issues list                   # issues by severity / dimension
datasentry score <run_id>                # six-dimension quality score
datasentry repair propose <issue_id> --file orders.csv   # fix proposal
datasentry drift latest orders           # drift between the two latest scans
datasentry-server                       # Web UI + REST API at http://localhost:8000

Scan a DuckDB file (optional — any CSV/Parquet/JSONL/XLSX/SQLite works):

datasentry scan analytics.duckdb --table payments
datasentry scan analytics.db --table payments     # SQLite (Step 54)

Scan a MySQL table (V5, Step 56 — via DuckDB mysql extension, no client library; --table required) or a cloud file (V5, Step 57 — CSV/Parquet/ JSONL over s3:// gs:// az://, credentials from process env / secrets):

datasentry scan "mysql://user:pass@localhost:3306/analytics" --table payments
datasentry scan s3://bucket/orders.csv            # AWS credentials from env

Scan a PostgreSQL table (V4, Step 55 — DSN is passed on the command line / via DATASENTRY_PG_DSN and is never persisted or logged):

datasentry scan "postgresql://user:pass@localhost:5432/analytics" --table payments
DATASENTRY_PG_DSN="postgresql://user:pass@localhost:5432/analytics" \
  datasentry scan postgresql:// --table payments --schema public

Credentials (V5, Step 59 — ADR-059)

connection_ref resolution chain: process environment variable, then ~/.config/datasentry/secrets.env (overridable via DATASENTRY_CONFIG_HOME or XDG_CONFIG_HOME), then DataSourceNotFoundError:

datasentry secrets set DATASENTRY_PG_DSN      # interactive, no echo, chmod 600
datasentry secrets list                       # key names only (audit-safe)
datasentry secrets get DATASENTRY_PG_DSN
datasentry secrets rm DATASENTRY_PG_DSN

The secrets file uses KEY=VALUE lines (env-var-shaped keys, source-able); the directory is 0700 and the file 0600 — both enforced on read and write. Credentials never enter scan runs, logs, reports, or webhook payloads; all connector errors are redacted (postgresql://*** / passwd=***).

Contract-driven scanning (optional):

datasentry contract validate contract.yaml
datasentry contract export contract.yaml --as pandera   # or --as ge
datasentry scan orders.csv --contract contract.yaml     # gate + rules bound

Architecture

flowchart LR
    subgraph Sources
        CSV[CSV / Parquet / JSONL / XLSX] --> Exec[DuckDB SQL executor]
        DDB[(.duckdb / .db files)] --> Exec
        PG[(PostgreSQL / SQLite)] --> Exec
    end
    Exec --> Dets[39 detectors]
    Dets --> Fuse[Evidence fusion]
    Fuse --> Score[Six-dimension scoring]
    Score --> Gate[Quality gate]
    Gate --> Report[JSON / MD / HTML / JUnit / SARIF]
    Report --> UI[Web UI + trends]
    Report --> MCP[MCP stdio server]
    Report --> CLI[CLI / REST]
    subgraph AI
        LLM[LLM provider: OpenAI / Ollama]
        Red[PII redaction]
        Audit[llm_cache + audit]
        LLM --> Red
        Red --> Rules[NL → rule candidates]
        Rules --> Repair[AI repair candidates]
        Audit -.->|every call| Rules
    end
    Repair --> RepairEngine[Repair engine: propose → preview → apply → rollback]

• Local-first: DuckDB executes everything; LLM is optional (auto-degrades when unconfigured) and can run on local Ollama so data never leaves the machine.
• Deterministic core: detectors, scoring and repair are pure statistics — no AI guesswork in detection.
• Human in the loop: rules and repairs are proposals until you approve them; every repair is fingerprinted and rollback-able.

Features

Documentation

Development

uv sync
make check          # ruff + mypy --strict + pytest with 85% coverage gate
make demo           # M9 demo script
make bench          # 1e6-row benchmark (60s gate)
make build          # build both wheels (datasentry + datasentry_core)

Requirements: Python ≥ 3.12, uv. CI validates lint, types, coverage, demo, benchmark, API/UI smoke and wheel installability on every push.

Contributing

• Report issues with the exact data shape (or a minimal CSV) and the command you ran.
• Code: add a detector → register it in build_initial_detectors → cover it in tests/ → make check.
• Every change should reference its ADR decision; see docs/DEVELOPMENT.md for conventions.
• Please keep the human-in-the-loop invariant: anything AI proposes must remain a proposal until a human approves it.

License

Apache-2.0 — see LICENSE.