Datalens.ai 🔍

Data health intelligence that learns from your data, and tells you what to do next.

Learns normal. Flags what matters.

Point Datalens at files, MongoDB, BigQuery, S3 or an HTTP API. It profiles every object and field, scores quality with formulas you can read, finds keys, foreign keys and orphans, masks PII, learns what normal looks like for your feed, checks it against the schema you expect, and turns every finding into a prioritised action — in one offline HTML report, a JSON summary for CI/CD, and an optional chat you can ask follow-up questions.

The Verdict for day 1 of the bundled cricket demo: 87 / 100, Healthy, with the 3 fixes worth making first. Day 2's injected problems turn the same screen to At risk and name each one (see below).

Get started in 2 minutes

pip install datalens-ai
datalens analyze --source file --path your_data.csv      # no config needed; report in output/<name>_<tag>/

To reuse settings and keep credentials out of your commands, set up a config folder once:

datalens init                                             # ~/.datalens: config.yaml, secrets.yaml, .env, connections/
datalens connection-new orders --source file --path ./exports   # or mongodb, http, s3, bigquery
datalens analyze --cc orders

Keep config anywhere (a server, a shared repo) with --config-dir /path or DATALENS_CONFIG_DIR, and run datalens config-show to see what's in use.

📘 Setup Guide (with Setup Connections) · ❓ FAQ: sample size, full scans, output folder, history, how drift works, CI/CD.

Contents

• Get started in 2 minutes
• Day 1 vs day 2 in 60 seconds
• It learns what normal looks like
• It tells you what to do next
• Bring your own schema (BYOS)
• Every number explains itself
• Wire it into CI/CD
• AI review and chat
• Coming from variety.js?
• Install · Examples · Docs · FAQ

Day 1 vs day 2 in 60 seconds

The repo ships a real, related dataset: IPL cricket from Cricsheet — five entities (matches, deliveries, players, teams, venues), ~160K rows, nested objects and arrays. Day 2 is the next load with 13 deliberate problems injected.

datalens analyze -s file -p test_data/cricket/day1 --pattern "*.jsonl.gz" -o output/demo --version-tag day1
datalens analyze -s file -p test_data/cricket/day2 --pattern "*.jsonl.gz" -o output/demo --version-tag day2 --detect-drift

Along the way it also finds real issues in the source: season is an integer for 2017–2019 and a string ("2020/21", "2022") afterwards — across five fields in three objects, reported as one root cause.

It learns what normal looks like

Real feeds wobble. In IPL, most days have one match (~250 deliveries) and some have two (~500). Fixed day-over-day thresholds cry wolf on every double-header; ignoring volume misses a truncated load.

With --compare-to rolling, each metric — row counts, coverage of every field, category counts, orphan rates, every score — learns its own normal range from recent runs:

M = median(last N values)      MAD = median(|x − M|)      σ̂ = 1.4826·MAD   (robust: one bad day can't inflate it)
band = [min(M − 3σ̂ₑ, min seen·0.95),  max(M + 3σ̂ₑ, max seen·1.05)]      σ̂ₑ = max(σ̂, 5%·|M|, sampling noise)

Recurring patterns are learned as normal, runs that breached are kept out of the baseline, and values seen anywhere in the window aren't "new". Replaying 12 real match days plus one broken load (feed truncated, 30% nulls):

Try it: examples/scenarios/05-rolling-daily-baseline. Prefer fixed rules? Every threshold is configurable per dataset, object and field, with separate drop and increase limits and your own messages — see Drift rules.

It tells you what to do next

The Action Plan turns every finding — drift, expected-schema failures, PII, orphans, mixed types, stale data, AI suggestions — into one action with the evidence, why it matters, the affected fields and a copy-ready fix. Related findings are merged (the rename explains "required field missing"), priorities come from severity and breadth, and you can tick items off and export the plan as CSV, Markdown or JSON.

Bring your own schema (BYOS)

Tell Datalens what the data should look like with a JSON Schema — required fields, types, enums, ranges, patterns — and put thresholds right next to the fields:

{"title": "orders", "type": "object", "required": ["id", "title"],
 "x-datalens": {"defaults": {"coverage": {"change_pct": 20}}},
 "properties": {
   "id":      {"type": "string", "x-datalens": {"coverage": {"drop_pct": 5}}},
   "title":   {"type": "string", "x-datalens": {"coverage": {"drop_pct": 10}}},
   "country": {"type": "string", "x-datalens": {"coverage": {"change_pct": 20}}}}}

datalens schema infer -s file -p data/ -o expected.json     # no schema yet? learn one from a good load, then edit it
datalens analyze -s file -p data/ --schema expected.json --detect-drift

Failures show up in Health → Expected Schema, in the Action Plan, in the health score and in the CI exit code.

Every number explains itself

Hover any ⓘ for what a metric means and how it's calculated. Verdict → How scores work shows this run's health score step by step and the DQI of every object and dimension with its weight. The same definitions are in docs/METRICS.md and in the terminal (datalens glossary dqi), generated from the code's own constants so they can't disagree.

Wire it into CI/CD

Everything in the report is available from the CLI:

datalens analyze --cc nightly_export -o output --version-tag "$(date +%F)" --run-date "$(date +%F)" \
  --compare-to rolling --schema expected.json \
  --fail-on fail --min-score health=70,dqi=85 --max-drop dqi=5 \
  --notify "slack:$SLACK_WEBHOOK" --format json > summary.json

Exit codes: 0 ok · 1 warnings (with --fail-on warn) · 2 breach or failed gate · 3 error. Also: datalens scores, datalens drift (re-check from history without reading data), datalens history list. A ready GitHub Actions workflow is in examples/scenarios/08-ci-gate.

AI review and chat

Datalens is fully useful offline. Add --ai claude (logged-in Claude CLI), anthropic, openai, cursor or copilot (the model is auto unless you pin one with --ai-model; a rejected model falls back to auto) and a model reviews the findings — schema, keys, orphans, drift — and adds domain-aware recommendations to the Action Plan, marked AI — shown as scannable finding cards (gist, highlighted numbers, details on demand) in Verdict → AI Review. No sample values are sent; PII fields are flagged as masked.

Then ask follow-up questions:

datalens serve output/demo/day2_day2      # report + chat on http://127.0.0.1:8765
datalens ask "Which bowler ids are orphaned most often?" output/demo/day2_day2
datalens serve output/demo/day2_day2 --ai copilot --ai-model claude-opus-5   # any provider and model

The model can run read-only SQL on a PII-masked sample of the data; every query is shown with the answer. Answers can be downloaded (CSV/Markdown), added to the Action Plan, or the whole session exported. Localhost only, per-session token, and the report file is never changed.

Opened directly, a report shows a dimmed Ask Datalens · OFF button. Click it for the copyable serve command. Pick any provider and model with --ai and --ai-model: see scenario 13.

Coming from variety.js?

variety.js tells you which keys exist, with which types, in what share of documents. That's where Datalens starts:

datalens analyze --source mongodb --uri "$MONGO_URI" --db shop --collections orders,customers --detect-drift
datalens analyze --source bigquery --project my-proj --dataset shop --tables orders,customers --detect-drift

Install

pip install datalens-ai                 # core: files, MongoDB, HTTP
pip install "datalens-ai[ai]"           # + Anthropic / OpenAI SDKs (CLI logins need nothing extra)
pip install "datalens-ai[bigquery]"     # + Google BigQuery connector
pip install "datalens-ai[all]"          # everything

The command is datalens (datalens --help). From source, for development:

git clone https://github.com/m8d8/datalens.ai.git
cd datalens.ai
uv sync                      # or: pip install -e .
uv sync --extra ai           # optional: Anthropic / OpenAI SDKs (CLI logins need nothing extra)
uv sync --extra bigquery     # optional: Google BigQuery connector
uv sync --extra dev          # tests: uv run pytest

Quick start on your own file:

uv run datalens analyze --source file --path data.csv --out-dir output

Each run writes to output/<source>_<tag>/: the HTML report, a run summary (JSON), the drift report and expected-schema check (when used), the masked schema JSON, a markdown summary and, with --ai, the AI review. Reusable connections with secrets kept out of git: Setup Guide and Setup Connections.

Library use:

from datalens import analyze

result = analyze({"source": "file", "path": "data.csv"})
result.decision["health_verdict"]   # {'status': 'attention', 'score': 79.2, 'drivers': [...], 'penalties': {...}}
result.decision["next_steps"]       # every action with evidence and fix
result.drift_report                 # when a reference run is given

Examples for every scenario

Reproduce every run and screenshot in this README: AI=claude bash examples/demo/run_demo.sh.

Documentation

Data attribution

The demo data in test_data/cricket/ is derived from Cricsheet, available under the Open Data Commons Attribution License. Day 2 and the last daily load contain deliberately injected problems and synthetic email addresses — see test_data/cricket/ATTRIBUTION.md. Regenerate with python examples/demo/build_demo_data.py.

License

MIT License — see LICENSE.