DataSentry
Evidence-driven, local-first AI copilot for data quality.
Detect · Explain · Validate · Repair — with statistical evidence, AI assistance, and human approval.
中文导读：DataSentry 是一个以统计证据为基础、以 AI 为辅助、以人工审批为保障的本地优先数据质量平台。 一次扫描生成六维质量评分，每个问题带证据链；自然语言即可提出规则与修复方案，但只有人工批准才生效。 数据不出机器（LLM 可接本地 Ollama），DuckDB 执行引擎，百万行 10 秒级。
Try it in 3 commands
pip install datasentry-ai
datasentry scan orders.csv # detect → fuse → score → persist
datasentry issues list --severity high # each issue with samples + ratios + confidence
What is DataSentry?
DataSentry scans your data (CSV / Parquet / JSONL / XLSX / DuckDB / SQLite / PostgreSQL / MySQL / cloud objects on s3:// gs:// az://) and produces:
- 39 evidence-driven detectors — missingness, dates, encodings, cross-field rules, cross-table foreign keys, duplicates, outlier models — every issue carries samples, ratios, and confidence.
- Six-dimension quality score — completeness, validity, uniqueness, consistency, integrity, timeliness, with explainable weights.
- Repair loop with human approval — propose → preview → apply (copy + rollback artifact) → verify → rollback. AI suggests; you decide.
- Drift engine — compare historical scans: schema, row-count, score and issue-distribution drift.
- Quality gates in CI —
scan --fail-onblocks releases; export JSON / Markdown / HTML / JUnit / SARIF. - LLM assistance, safely — NL → rule/repair candidates with preflight; PII redacted into an encrypted vault; every call audited.
- Cron scheduling + distributed execution — SQLite job queue, webhooks, quality gates, change-aware skip;
datasentry workernodes with pool routing, failover and parallel dispatch. - Multiple surfaces — CLI, REST API, Web UI with trends, and an MCP stdio server (24 tools) for LLM agents.
Live demo report — orders-report.html (200 rows with 15 injected quality issues)
Quick start
pip install datasentry-ai # or: uv sync (source checkout)
datasentry # interactive terminal UI (TUI): dashboard / scan / issues / repair
datasentry scan orders.csv # detect → fuse → score → persist
datasentry scan "a.csv, b.csv, data/*.csv" # batch scan (comma/newline separated, globs)
datasentry issues list # issues by severity / dimension
datasentry score # six-dimension score of the latest scan
datasentry repair propose <issue_id> --file orders.csv # fix proposal (apply = copy + rollback artifact)
datasentry repair verify <run_id> # re-scan the repaired copy; CI gate (exit 0 = no regression)
datasentry drift latest orders # drift between the two latest scans
datasentry-server # Web UI + REST API at http://localhost:8000
Every CLI command stays available for scripts and CI; datasentry scan streams
live detector progress to stderr so stdout stays clean JSON. Connectors include
DuckDB/SQLite tables (--table), MySQL, PostgreSQL (DSN via env or secrets,
never logged), and cloud objects (credentials from env / secrets).
Architecture
flowchart LR
subgraph Sources
CSV[CSV / Parquet / JSONL / XLSX] --> Exec[DuckDB SQL executor]
DDB[(.duckdb / .db files)] --> Exec
PG[(PostgreSQL / SQLite / MySQL / cloud)] --> Exec
end
Exec --> Dets[39 detectors]
Dets --> Fuse[Evidence fusion]
Fuse --> Score[Six-dimension scoring]
Score --> Gate[Quality gate]
Gate --> Report[JSON / MD / HTML / JUnit / SARIF]
Report --> UI[Web UI + trends]
UI --> Compare[Run compare: dimension / severity / issue-level diff]
UI --> BatchWeb[Batch scan: glob + comma paths, live progress]
Report --> MCP[MCP stdio server]
Report --> CLI[CLI / REST]
CLI --> BatchCLI[Batch scan: comma + newline + glob, per-file summary]
TUI[Terminal UI] --> BatchTUI[Batch scan + issue center + repair workflow]
subgraph Scheduling
Q[(SQLite job queue)] --> Sched[Scheduler + worker thread]
Sched -->|dispatch| Pool[Worker pool: round-robin + failover + parallel]
Pool --> W1[Worker A: /rpc/execute]
Pool --> W2[Worker B: /rpc/execute]
end
subgraph AI
LLM[LLM provider: OpenAI / Ollama]
Red[PII redaction + encrypted vault]
Audit[llm_cache + audit]
LLM --> Red
Red --> Rules[NL → rule candidates]
Rules --> Repair[AI repair candidates]
Audit -.->|every call| Rules
end
Repair --> RepairEngine[Repair engine: propose → preview → apply → rollback]
- Local-first: DuckDB executes everything; LLM is optional (auto-degrades when unconfigured) and can run on local Ollama so data never leaves the machine.
- Deterministic core: detectors, scoring and repair are pure statistics — no AI guesswork in detection.
- Human in the loop: rules and repairs are proposals until you approve them; every repair is fingerprinted and rollback-able.
The repair loop (scan → propose → apply → verify → rollback)
Available identically on CLI, Web UI, REST and MCP — batchable, audit-friendly, and the
source file is never overwritten (each apply writes a repaired copy + before snapshot):
datasentry repair propose-batch <run_id> --file data.csv --all # read-only
datasentry repair apply-batch <run_id> --file data.csv --all # issues without a proposal skip, not fail
datasentry repair verify <run_id> # re-scan copy: fixed/persistent/new types
datasentry repair diff <run_id> # changed rows: line + col: old -> new
datasentry repair rollback <run_id> # restore the snapshot
Batch commands exit 0 on full success, 4 on partial failure (each failure under
errors in the JSON envelope — add --format json to machine-read it).
Verify is the gate: repair verify exits 0 unless the repair introduced a regression
(--require-clean demands zero remaining issues); the Web artifact page
(/ui/repairs/{id}/artifact) shows the before/after row diff, and the compare view
links every FIXED group back to the repair that fixed it. Same report over REST
(POST /repairs/{id}/verify, GET /repairs/{id}/diff) and MCP (repair_verify).
Features
| Area | What you get |
|---|---|
| Detection | 39 detectors across 6 dimensions; SQL-pushdown single-table; plugin API (plugins/ auto-load, SHA-256 integrity locks) |
| Scoring | 0–100 six-dimension score, severity normalization, contract criticality |
| Contracts | YAML contract DSL → validation + gate + Pandera / Great Expectations export |
| Repair | trim / normalize case / replace missing token / set null / clip values; preview re-runs rules; verify + diff on 4 surfaces |
| Drift | schema / row-count / score / issue-distribution signals between historical scans |
| AI | NL→rules with preflight + approval gate; AI repair candidates with locked operation surface; PII vault + key rotation |
| Scheduling | cron jobs, manual triggers, run history, webhooks, quality gates, change-aware skip |
| Distributed | datasentry worker nodes; pool routing with failover + cooldown + health checks; parallel dispatch |
| Interfaces | CLI · REST API · Web UI (/ui, /ui/scans, /ui/trends, /ui/compare, /ui/repairs) · MCP stdio (24 tools) |
| Engineering | 11-stage CI, wheel build + isolated install smoke, 1e6-row benchmark gate |
Documentation & blog
- docs/DEVELOPMENT.md — development notes, conventions
- docs/00-设计裁决记录-ADR.md — 110+ ADRs
- Detect → fix → verify: the data quality loop / 中文版
- Verifying the fix: four ways to prove a repair worked / 中文版
Development
uv sync
make check # ruff + mypy --strict + pytest with 85% coverage gate
make demo # demo script
make bench # 1e6-row benchmark (60s gate)
make build # build both wheels (datasentry + datasentry_core)
Contributing
- Report issues with the exact data shape (or a minimal CSV) and the command you ran.
- Code: add a detector → register it in
build_initial_detectors→ cover it intests/→make check. - Please keep the human-in-the-loop invariant: anything AI proposes must remain a proposal until a human approves it.
License
Apache-2.0 — see LICENSE.
1 maintainer
Release files for datasentry-ai 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| datasentry_ai-1.0.0.tar.gz | 1.5 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datasentry_ai-1.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.6 MB
Release files / datasentry_ai-1.0.0.tar.gz
| Download URL | datasentry_ai-1.0.0.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d7561311954f812fd882b742f2e6173cffeeead398e7cbe1158af898621ce3fd
|
|
BLAKE2b-256 checksum How to use checksums |
4a01cab2115af8149a85885f8cd9fc5821654f435f2e55b0704a819343bec427
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 20, 2026.
Transparency logRelease files / datasentry_ai-1.0.0-py3-none-any.whl
| Download URL | datasentry_ai-1.0.0-py3-none-any.whl |
|---|---|
| Size | 130.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
29bbedf4827a619cc2d1b3c018306d48bd7e859da255b0e174428f5a4c196362
|
|
BLAKE2b-256 checksum How to use checksums |
3c2a07effd5c5acbb6f681bebc1a9c688b94958a26a029d9e0cedcc6c129ef60
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 20, 2026.
Transparency log
