data-quality-agent-dk 0.1.0

Agentic Data Quality Pipeline: profiles CSVs, proposes DQ rules, writes+validates SQL to check them, and reports confirmed violations.

data-quality-agent-dk

Agentic data-quality pipeline. Point it at a folder of CSVs; it profiles every table, proposes DQ rules from real evidence, writes + validates the SQL to check each one (read-only, sandboxed, retried on failure), and hands back a JSONL report plus an optional Streamlit dashboard.

Step 1 — Set up your environment

Create a .env file in your project's working directory (not inside this package) with whichever credentials your chosen models need. At minimum, the LLM provider key(s) for whatever you pass as planning_model_provider / worker_model_provider:

pick whichever provider(s) you're using

GOOGLE_API_KEY=... ANTHROPIC_API_KEY=... OPENAI_API_KEY=... OPENROUTER_API_KEY=...

optional: enables per-table LangSmith tracing (profiling -> rule planning -> every SQL generate/execute retry, as one trace per table)

LANGCHAIN_TRACING_V2=true LANGCHAIN_API_KEY=... LANGCHAIN_PROJECT=your-project-name

You only need the key(s) matching the provider(s) you actually pass in — if both planning_model_provider and worker_model_provider are "google_genai", you only need GOOGLE_API_KEY.

Step 2 — Install

pip install data-quality-agent-dk[google]      # Gemini
pip install data-quality-agent-dk[anthropic]   # Claude
pip install data-quality-agent-dk[openai]      # OpenAI
# any other provider langchain's init_chat_model supports: install its
# integration package yourself, then pass provider="..." below.

Step 3 — Run it

from data_quality_agent_dk import run_pipeline

result = run_pipeline(
    data_dir="data",              # folder of .csv files, one table per file
    output_dir=".",               # where dq_report.jsonl / todo_list.md land

    # any model string + provider langchain's init_chat_model accepts
    planning_model="gemini-2.5-flash",
    planning_model_provider="google_genai",
    worker_model="gemini-2.5-flash-lite",
    worker_model_provider="google_genai",

    temperature=1.0,
    max_tokens=40000,

    sample_rows_limit=100,
    max_retries=3,
    max_violation_rows_shown=3,
    suspicious_violation_ratio=0.45,

    display=True,   # launch the Streamlit dashboard when the run finishes
)

print(result["report_path"], result["failed_tables"])

data_dir is the only required argument. Everything else has a default (see the table below) — but planning_model/planning_model_provider and worker_model/worker_model_provider default to Gemini models, so if you're using a different provider, you must set those four explicitly, or the pipeline will try to call Gemini regardless of which API key you've set.

Choosing your own values

Every field below has a working default — you only need to touch a value if your data or budget genuinely calls for something different.

Parameter	Ask yourself	Default
planning_model	This makes one exhaustive, column-by-column reasoning pass per table — is my data complex enough (many columns, many edge cases) to justify a stronger/pricier model here?	gemini-2.5-flash
worker_model	This is called once per rule — PK inference, every SQL write/repair, every report entry. It runs many times per table, so cost and latency matter more than raw reasoning depth here. Do I want the cheapest/fastest model that's still reliable?	gemini-2.5-flash-lite
temperature	Do I want the same table to produce roughly the same rules and SQL if I re-run it? Lower = more deterministic/reproducible. Higher = more variation between runs.	1.0 (gemini supports this better)
max_tokens	How wide are my tables, and how long do my rule descriptions/SQL tend to get? A table with 40+ columns or many cross-column rules needs more headroom than a 5-column table.	40000
sample_rows_limit	How big is my dataset, and how representative does a sample need to be to catch rare-but-real issues? A 5000-row table can be mostly sampled; a 5-million-row table needs a bigger absolute sample to surface low-frequency problems, even though it's a smaller fraction of the whole.	100
max_retries	How much do I value catching every possible rule vs. how much do I care about runtime/cost? Higher retries recover more rules from a bad first attempt, at the cost of more LLM calls per rule that's genuinely hard to express in SQL.	3
max_violation_rows_shown	Am I using this report for a human to skim, or feeding it into another system? Fewer sample rows keeps the report/dashboard readable; more gives more diagnostic context per issue.	3
suspicious_violation_ratio	How dirty do I expect this data to genuinely be? Raise it if you have reason to expect genuinely high violation rates.	0.45
display	Toggling to true allows you to view the output in a simple streamlit interface	True

For the full technical reference — every config field, the graph architecture, and troubleshooting — see docs/config.md.