Skip to main content

Agentic Analytics

How often does agentic analytics get the answer right, by question type and by platform, and which fixes close the gap? agentic-analytics asks plain-language questions of your data through AI agents and scores every answer against gold SQL, with no LLM judge.

Chart: accuracy of every configuration with its 95% interval, the strict score beside it, and the run's caveats; the figures are in the text below.

Claude (MCP), GPT (MCP) and Cortex Analyst each answered 100 questions from the uk_price_paid@1.0 pack 3 times (data as of 2026-08-31), and every answer was scored without an LLM judge: its SQL result was compared with gold SQL whose results agree on ClickHouse and Snowflake (cross-engine gold), and on ambiguous questions the backend had to ask back. Baseline accuracy was 76.0% (95% CI 67.7–84.0%; strict 76.0%) for Claude (MCP), 97.0% (95% CI 93.0–100.0%; strict 92.0%) for GPT (MCP) and 85.0% (95% CI 78.0–92.0%; strict 62.0%) for Cortex Analyst; the largest fix effect above noise was examples on Claude (MCP), +15.7 pts (95% CI +8.3 to +23.3), and descriptions lowered GPT (MCP)'s accuracy, −5.0 pts (95% CI −9.0 to −2.0). The 95% intervals are two-level bootstrap intervals over questions and trials and are conservative by design (about 99% coverage in simulation), so overlapping intervals do not show a difference; the strict score is the pre-registered one, before the four narrow shape equivalences the headline accepts for every backend (a percentage given as a fraction, one pivoted row, a zero-count check for no data, or a period given as a date).

Walkthrough: the headline chart, the dashboard, the uplift from each fix, accuracy by category, the safety probes, the CLI, the demo's saved answers and passcode mode, and the case study

Quickstart · Case study · Results · Screenshots · Step-by-step guide · Decision log · Measurements · EvalKit

What it is

The bundled benchmark asks questions about UK house sales (HM Land Registry Price Paid Data) in two ways over the same data:

  • An agent with tools: Claude or GPT querying ClickHouse through ClickHouse's remote MCP server.
  • A warehouse-native assistant: Snowflake Cortex Analyst over a semantic view.

Every question has gold SQL for both engines, and the two results must agree before any run (cross-engine gold). Every answer's own SQL is run again and compared with the gold result using EvalKit's sql_match; ambiguous questions count as correct only when the backend asks back. Three fixes (column descriptions, worked example queries, a self-check) were frozen before the baseline and each was measured alone against it. Everything dataset-specific lives in a dataset pack, so the same harness runs on your own data.

Status

Built and measured. The baseline and the fix runs are done and scored; the baseline was hand-checked by the builder and the fix runs were checked by an independent verifier. The headline, the results tables and the charts are generated from the committed published run summary, and each bullet below cites the record it comes from. Released as v0.1.0; what is live, and what comes after it:

  • On PyPI as v0.1.0: pip install agentic-analytics (release record: the release workflow published it by trusted publishing, then installed and ran it on Linux, macOS and Windows).
  • The results dashboard is live at biplob819.github.io/agentic-analytics (deploy record). The hand-run Pages workflow builds the dashboard from the committed published-run record, so the public page has every figure but no per-answer records or traces (its failure explorer is empty; the full dashboard comes from agentic-analytics report on a run folder).
  • The hosted demo is live at agenticanalytics12.streamlit.app on Streamlit Community Cloud. It replays saved answers by default; the live, passcode-gated mode answered 20 timed questions with a p95 of 14.1 s and no timed-out or errored panel (latency record, DEM-15).
  • Monitoring runs by hand. The 12-question smoke run with its accuracy gate is a hand-run workflow; its schedule stays off until the owner turns it on (OWNER D-103).
  • Run history, trend and drift come with EvalKit v0.4.0 (v0.1.1). v0.1.0 ships on EvalKit 0.3.0. From v0.1.1 every run appends the metrics of the configurations it newly asked, and every reconcile its reconciled costs, to EvalKit's run ledger (docs/measurements/runs.jsonl), the dashboard links a trend page of accuracy, cost per correct answer and p95 latency with alarm bands, and the hand-run monitor checks each smoke against its band (MON-07, OWNER D-115, D-121).

Live build status: uv run python scripts/status.py.

Results

Configuration Accuracy (95% CI) Strict Asked back on ambiguous questions Asked back on answerable questions p95 latency Cost per correct answer
Claude (MCP) baseline 76.0% (67.7–84.0%) 76.0% 2.1% 0.0% 9.5 s $0.016
Claude (MCP) + descriptions 84.3% (77.3–90.7%) 83.3% 43.8% 0.0% 8.0 s $0.012
Claude (MCP) + examples 91.7% (86.0–96.7%) 89.7% 62.5% 0.0% 6.8 s $0.01
Claude (MCP) + self-check 74.0% (65.0–82.3%) 74.0% 0.0% 0.0% 8.0 s $0.015
GPT (MCP) baseline 97.0% (93.0–100.0%) 92.0% 81.2% 0.0% 9.6 s $0.0048
GPT (MCP) + descriptions 92.0% (86.7–96.3%) 88.7% 70.8% 0.0% 13.5 s $0.006
GPT (MCP) + examples 96.0% (92.0–99.0%) 93.7% 83.3% 0.0% 6.9 s $0.0036
GPT (MCP) + self-check 95.7% (91.7–99.0%) 91.0% 75.0% 0.0% 8.5 s $0.0049
Cortex Analyst baseline 85.0% (78.0–92.0%) 62.0% 6.2% 0.0% 7.5 s¹ $0.24
Fix Backend Change in accuracy (95% CI) Verdict after a Holm correction
descriptions Claude (MCP) +8.3 pts (+2.7 to +14.3) ▲ above noise, helps
examples Claude (MCP) +15.7 pts (+8.3 to +23.3) ▲ above noise, helps
self-check Claude (MCP) −2.0 pts (−4.3 to +0.0) ○ within noise
descriptions GPT (MCP) −5.0 pts (−9.0 to −2.0) ▼ above noise, hurts
examples GPT (MCP) −1.0 pts (−3.7 to +0.7) ○ within noise
self-check GPT (MCP) −1.3 pts (−3.0 to +0.0) ○ within noise

Claude (MCP) is claude-sonnet-5-5 and GPT (MCP) is gpt-6.1-sol, each querying ClickHouse through its remote MCP server. Run rescore-20261117T122217Z-6ccebe: pack uk_price_paid@1.0, data as of 2026-08-31, each question asked 3 times; costs reconciled with the warehouse's metered credits. Strict is the pre-registered score; the headline also accepts a percentage given as a fraction, one pivoted row, a zero-count check for no data, or a period given as a date (OWNER D-81), for every backend alike. The run of all fixes together is recorded in the dashboard but is not a headline figure. ¹ Snowflake account in AWS ap-southeast-7 (Thailand); Cortex inference crosses to the US (cross-region inference to AWS US regions, OWNER D-62), so Cortex latency includes that distance.

What the records show:

  • Most misses were guesses on ambiguous questions. Most of the baseline failures that an independent agent labelled by hand, reading each record, its SQL and the gold, were missed clarifications, the only cause for GPT and Cortex Analyst; Claude's other misses mostly came from ClickHouse's approximate median() against an exact gold median (failure taxonomy).
  • The strict score moves most for Cortex Analyst, which returned every percentage as a fraction and some periods as dates; the headline accepts those shapes for every backend, and the strict, pre-registered score stays beside it (OWNER D-81).
  • Worked examples helped Claude mostly by getting it to ask back on ambiguous questions, which no fix text mentions; descriptions lowered GPT's accuracy (fix runs; v0.1.0's record: fix runs on EvalKit 0.3.0).
  • The self-check fix never fired. Its frozen rules flagged none of the agents' final answers, so its rows measure trial-to-trial noise, not a fix.
  • The fixes made Claude easier to steer. With descriptions or examples, Claude followed one injected instruction in every trial; on the baseline it never did. No write probe got a write through on any backend (baseline).

Chart: change in accuracy for each fix against the same backend's baseline, in percentage points with 95% intervals and the Holm-corrected verdict in words; the figures are in the table above.

Chart: baseline accuracy by question category for each backend, with 95% intervals and the strict score as a hollow mark; few questions per category, so it is a map, not a ranking.

The records behind every figure: baseline, fix runs (v0.1.1, on EvalKit 0.4.0: one GPT trial moves, OWNER D-118; v0.1.0's fix runs), failure taxonomy, Snowflake metered credits, EvalKit parity (v0.1.0's on EvalKit 0.3.0: parity), the run history and the spend ledger, one line per paid run.

Screenshots

The results dashboard (agentic-analytics report): the run's stamp and caveats, accuracy with intervals and the strict score beside it, then fix uplift with the Holm verdict, accuracy by category, cost and clarification, safety probes and a failure explorer.

Screenshot of the results dashboard for the published run: the run's stamp (pack, snapshot, as-of date, gold check, cross-engine gold, cost reconciled) and licence credit, then the overview table with accuracy, strict accuracy and pass-all-trials, each with its 95% interval, for the Claude baseline and its single-fix configurations.

The demo page (demo/app.py), replaying saved answers from the benchmark run with no keys and no connection: pick a sample question and see the Claude agent's and Cortex Analyst's answer, SQL and rows side by side.

Screenshot of the demo page in saved-answer mode: a banner saying live querying is off, the licence credit, a sample-question picker, and the Claude with ClickHouse MCP and Snowflake Cortex Analyst panels, each with its saved answer, SQL, cost and time.

The passcode and own-key fields are always shown. Live answers need the owner's passcode and the owner turning that backend's compute on; until then the page says live compute is off and connects to nothing (OWNER D-60).

Screenshot of the demo page's ask-your-own mode with live compute off: the notice that no question is asked live until the owner turns it on, the own-key notice, and the passcode, question and optional Anthropic key fields.

The command line, with no keys and no accounts: init, pack check, report --example and the cost estimate of run --dry-run, then doctor, which lists the variables it looks for by name only and skips the engine checks it has no credentials for.

Terminal screenshot: agentic-analytics init writes run.yaml and .env.example, pack check validates the uk_price_paid pack, report --example writes the synthetic example dashboard, and run --dry-run prints the cost estimate and makes no calls.

Terminal screenshot: agentic-analytics doctor with no keys lists every environment variable by name as missing, never a value, and skips the ClickHouse, MCP role and Snowflake checks.

Step-by-step guide

1. Install. Python 3.11 or newer:

pip install agentic-analytics

Or from a clone with uv, to change the code or run the tests:

git clone https://github.com/biplob819/agentic-analytics
cd agentic-analytics
uv sync

The command is agentic-analytics, with the short alias aa (prefix with uv run inside the clone).

2. Look around with no keys.

agentic-analytics init                # writes run.yaml and .env.example into this folder
agentic-analytics pack check          # validates the bundled UK pack: schemas, lints, quotas
agentic-analytics report --example    # renders a dashboard from a bundled SYNTHETIC run

report --example writes a self-contained index.html you can open in a browser. Every figure on it comes from generated data, and the page says so.

3. Set up the engines. One engine is enough: with only ClickHouse or only Snowflake the run works and is stamped single-engine gold. Copy .env.example to .env first; the setup commands fill it in.

agentic-analytics setup clickhouse    # creates or adopts the ClickHouse Cloud service, MCP key and read-only user
agentic-analytics setup snowflake     # makes key pairs and prints the ACCOUNTADMIN SQL; --run runs it

Add the model keys you use (ANTHROPIC_API_KEY, OPENAI_API_KEY) to .env. Agents only ever reach the warehouses through read-only roles.

4. Check everything.

agentic-analytics doctor              # which variables are set (names only), engine reach, read-only roles

5. Load the data and check the gold.

agentic-analytics load                # loads the engines and checks they hold the same data
agentic-analytics gold                # runs every gold query on both engines; a run refuses to start without a clean check

6. Run. Every run prints its cost estimate first and stops at --max-cost.

agentic-analytics smoke               # 12 questions on every configured backend
agentic-analytics run --dry-run       # the cost estimate; makes no calls
agentic-analytics run --backend all --fix none --trials 3 --max-cost 25
agentic-analytics run --backend claude_mcp,gpt_mcp --fix descriptions --trials 3
agentic-analytics run --rescore       # rescore cached answers with no calls

7. Report.

agentic-analytics report --run latest                  # the dashboard, written into the run folder
agentic-analytics reconcile --run latest --credits credits.json   # metered warehouse credits replace the estimate

8. Your own data. Write a dataset pack (load SQL per engine, parity fingerprints, semantic model, questions with gold SQL for both engines, probes, fix content), check it, and point any command at it:

agentic-analytics pack check --pack path/to/my_pack
agentic-analytics run --pack path/to/my_pack --dry-run

uk_price_paid is the bundled reference pack; its files are a worked example of every part. The guides walk through writing a dataset pack and adding questions; reproducing the UK benchmark lists every step and record behind the published numbers.

How scoring works

  • No LLM judge. An answer-expected question is correct when the backend did not ask back, its final SQL executed, and the result matches the gold result (sql_match: extra columns are fine, missing or extra rows are not, numeric tolerance is set per question).
  • Strict beside the headline. The headline accepts four narrow shapes of a right answer for every backend: a percentage given as a fraction, one pivoted row, a zero-count check for no data, and a period given as a date. The strict, pre-registered score is shown beside it everywhere (OWNER D-81).
  • Clarification. An ambiguous question is correct only when the backend asks back. The rate of asking back on ambiguous questions is always shown next to the rate of asking back on answerable ones, so asking back on everything cannot look good.
  • Answers from the rows. Every number in an answer's text is checked against the executed result, and every agent query passes a read-only SQL guard before it reaches a warehouse.
  • Intervals. Accuracy carries 95% intervals from a two-level bootstrap over questions and trials. They are conservative by design: in simulation they covered the true value about 99% of the time, so two figures whose intervals overlap are not shown to differ. Per-category figures rest on few questions each and are a map, not a ranking.
  • Fixes. Each fix runs alone against the same baseline and counts as "above noise" only after a Holm correction. The run of all fixes together is recorded but is never a headline figure.
  • Safety probes. Injection and write probes run beside the questions; no write ever reaches a warehouse.

Cost, privacy and security

Model calls cost money on your keys; warehouse time and Cortex Analyst messages draw on your warehouse credits. Every run prints its estimate first, stops at --max-cost, and appends its spend to a ledger. There is no telemetry: data goes only to the model providers and warehouses you configure. Keys stay in your .env, and agents reach the data only through read-only roles. Details: controlling cost, the privacy statement (every destination, checked against the code) and the security policy (key handling, the read-only layers, private vulnerability reporting).

Licence

MIT licence for the code (LICENSE). The bundled data pack's credit, shown wherever its data appears:

Contains HM Land Registry data © Crown copyright and database right 2021. This data is licensed under the Open Government Licence v3.0.

Metadata

Release files for agentic-analytics 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentic-analytics 0.1.1
File Size Uploaded Provenance Actions
agentic_analytics-0.1.1.tar.gz 890.6 kB Without provenance Details for agentic_analytics-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentic-analytics 0.1.1
File Interpreter ABI Platform Actions
agentic_analytics-0.1.1-py3-none-any.whl Python 3 none any Details for agentic_analytics-0.1.1-py3-none-any.whl

Total release size: 1.5 MB

Release files / agentic_analytics-0.1.1.tar.gz

Download URL agentic_analytics-0.1.1.tar.gz
Size 890.6 kB
Tags Source
SHA-256 checksum
How to use checksums
22dbb8123221ca5ec3ceab170cdb0fbb35fd3248917e986ee1fe19a6efbcb1b1
BLAKE2b-256 checksum
How to use checksums
56f662fe5b6464959f54d98542f35e8eedb7c075f2d1326987f8fec555d3e26e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.21 {"installer":{"name":"uv","version":"0.11.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / agentic_analytics-0.1.1-py3-none-any.whl

Download URL agentic_analytics-0.1.1-py3-none-any.whl
Size 611.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7c6a3c0ba45c52631dcbb306cca8f0332f5cd86acb47489e0ad348aaaed26d40
BLAKE2b-256 checksum
How to use checksums
aeaf9ddc9e98edf6df23274c98a27576b67cf5db82015fc5615817440fa7177c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.11.21 {"installer":{"name":"uv","version":"0.11.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for agentic-analytics 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page