data-quality-agent-dk 0.1.2

Agentic Data Quality Pipeline: profiles CSVs, proposes DQ rules, writes+validates SQL to check them, and reports confirmed violations.

data-quality-agent-dk

Runs an Agentic Data Quality pipeline. Mandatory to point it to the folder of CSVs. Other parameters can be configured as shown below. Your tables are then profiled, potential DQ issue rules are created from real evidence, the SQl for each rule is then generated, validated and executed with a retry mechanism. At the end, a JSONL report plus an optional Streamlit dashboard displays the existing Data Quality issues in the files uploaded.

Requirements

• Python 3.10+
• An API key for at least one LLM provider (e.g. Gemini, Anthropic, OpenAI, OpenRouter)

Installation

Running just:

pip install data-quality-agent-dk

gets you the base package — langchain, langgraph, duckdb, streamlit, and everything else the pipeline itself needs. What it does not include is a library for talking to any specific LLM provider, since most people only use one and there's no reason to install libraries for providers you're not using.

So say you want to use Gemini: you add [google] to the install command, which tells pip "install the base package, and also install what's needed for Gemini support":

pip install data-quality-agent-dk[google]

That [google] is called an extra — an optional add-on dependency set. Under the hood it just pulls in langchain-google-genai alongside everything else. The same idea applies for the other built-in providers:

pip install data-quality-agent-dk[anthropic]   # Claude
pip install data-quality-agent-dk[openai]      # OpenAI

Using a provider that isn't one of these three (e.g. OpenRouter)? Any model + provider that langchain's init_chat_model supports will work — just install that provider's own package yourself and pass it as a parameter when calling run_pipeline (see Step 2 below).

Step 1 — Set up your environment

Create a .env file in your project's working directory (not inside this package) with whichever credentials your chosen models need. At minimum, the LLM provider key(s) for whatever you pass as planning_model_provider / worker_model_provider:

# pick whichever provider(s) you're using
GOOGLE_API_KEY=...
ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
OPENROUTER_API_KEY=...

# optional: enables per-table LangSmith tracing (profiling -> rule
# planning -> every SQL generate/execute retry, as one trace per table)
LANGSMITH_TRACING_V2=true
LANGSMITH_API_KEY=...
LANGSMITH_PROJECT=your-project-name

You only need the key(s) matching the provider(s) you actually pass in — if both planning_model_provider and worker_model_provider are "google_genai", you only need GOOGLE_API_KEY.

Step 2 — Run it

from data_quality_agent_dk import run_pipeline

result = run_pipeline(
    data_dir="data",              # folder of .csv files, one table per file
    output_dir=".",               # where dq_report.jsonl / todo_list.md land

    # any model string + provider langchain's init_chat_model accepts
    planning_model="gemini-3.5-flash",
    planning_model_provider="google_genai",
    worker_model="gemini-3.1-flash-lite",
    worker_model_provider="google_genai",

    temperature=1.0,
    max_tokens=40000,

    sample_rows_limit=100,
    max_retries=3,
    max_violation_rows_shown=3,
    suspicious_violation_ratio=0.45,

    display=True,   # launch the Streamlit dashboard when the run finishes
)

print(result["report_path"], result["failed_tables"])

What you get back

run_pipeline returns a dict:

Key	What it is
table_names	Every table (CSV) that was processed
failed_tables	Tables that errored out and were skipped
report_path	Path to dq_report.jsonl — one line per confirmed violation
todo_path	Path to todo_list.md — every rule considered, with its final status

If display=True (the default), a Streamlit dashboard also opens automatically once the run finishes, showing the same report visually with filters and sample violating rows per rule.

Choosing your own values

Every field below has a working default — you only need to touch a value if your data or budget genuinely calls for something different.

Parameter	Ask yourself	Default
planning_model	This makes one exhaustive, column-by-column reasoning pass per table — is my data complex enough (many columns, many edge cases) to justify a stronger/pricier model here?	gemini-3.5-flash
worker_model	This is called once per rule — PK inference, every SQL write/repair, every report entry. It runs many times per table, so cost and latency matter more than raw reasoning depth here. Do I want the cheapest/fastest model that's still reliable?	gemini-3.1-flash-lite
planning_temperature	Do I want the same table to get roughly the same proposed rule set if I re-run it? Lower = more deterministic/reproducible planning. Higher = more variation in which rules get proposed.	1.0
worker_temperature	Do I want the SQL for a given rule to come out the same on every retry/re-run? This is independent of planning_temperature — you can plan deterministically and still let SQL generation vary, or vice versa.	1.0
planning_max_tokens	How wide are my tables? A table with 40+ columns needs more headroom for the rule-planning call to return its full column-by-column reasoning and rule list without truncating.	40000
worker_max_tokens	How long do individual rule descriptions/SQL tend to get? This ceiling applies once per rule (PK inference, each SQL attempt, each report entry), so it rarely needs to be as high as planning_max_tokens unless your rules involve long, multi-condition SQL.	40000
sample_rows_limit	How big is my dataset, and how representative does a sample need to be to catch rare-but-real issues? A 5000-row table can be mostly sampled; a 5-million-row table needs a bigger absolute sample to surface low-frequency problems, even though it's a smaller fraction of the whole.	100
max_retries	How much do I value catching every possible rule vs. how much do I care about runtime/cost? Higher retries recover more rules from a bad first attempt, at the cost of more LLM calls per rule that's genuinely hard to express in SQL.	3
max_violation_rows_shown	Am I using this report for a human to skim, or feeding it into another system? Fewer sample rows keeps the report/dashboard readable; more gives more diagnostic context per issue.	3
suspicious_violation_ratio	How dirty do I expect this data to genuinely be? Raise it if you have reason to expect genuinely high violation rates.	0.45
display	Toggling to true allows you to view the output in a simple streamlit interface	True

Learn more

• docs/architecture_overview.md — how the pipeline works stage by stage, with a diagram of the per-table graph.
• docs/config.md — every config field, its default, and the common pitfalls people hit when overriding it.