Skip to main content

MindzEval

MindzEval proves your SQL agent is accurate and safe before it reaches users, and keeps proving it on every change.

Status: release 0.1.0. New here? Start with getting started. Feedback and bug reports are welcome in the issues.

What it does

  • Grades the data, not the SQL. The agent's results are compared with the results of a reference query, so correct SQL written a different way still passes.
  • Checks the answer key first. Reference queries are validated against the live database (empty, drift, unstable, NULL groups, orphan-inflated) before any agent is graded.
  • Scores behaviour both ways. Asking for clarification or declining is rewarded when it's right and penalised when it isn't.
  • Separates display limits from logic errors. The agent's SQL is re-run without its display LIMIT, so a UI row cap isn't mistaken for a wrong query.
  • Checks safety. Write attempts, forbidden tables and PII in results or answers.
  • Works with any agent. LangChain, LangGraph, CrewAI, LlamaIndex, the OpenAI Agents SDK, plain Python, or any HTTP service.
  • Compares models and prompts. Matrix runs evaluate every combination side by side.
  • Read-only by design. Every query MindzEval runs is parsed and refused unless it is a single read, then executed in a read-only transaction with a timeout.

Installation

MindzEval requires Python 3.11 or newer:

# SQLite only
pip install mindzeval

# With PostgreSQL support
pip install "mindzeval[postgres]"

Quickstart

mindzeval quickstart demo
cd demo
mindzeval run

The example agent fails some cases on purpose so you can see what each verdict means.

Your own agent

mindzeval.yaml:

project: shop-agent
agent:
  type: python
  target: my_project.eval_adapter:answer     # or type: http with a URL
database:
  url: postgresql://reader:${DB_PASSWORD}@db.internal/shop
datasets: cases/*.yaml

cases/customers.yaml:

- id: customers_in_germany
  question: Which customers are in Germany?
  reference_sql: SELECT id FROM customers WHERE country = 'DE'
  key: id

- id: vague_request
  question: Show me the data
  expect: {behavior: clarify}
mindzeval lint       # check config and cases (no connections, no secrets)
mindzeval validate   # check reference queries against the database
mindzeval run        # evaluate; exits 1 if thresholds aren't met

Documentation

Development

git clone https://github.com/MindzKonnected/mindzeval.git && cd mindzeval
python -m venv .venv && .venv\Scripts\activate      # macOS/Linux: source .venv/bin/activate
pip install -e ".[dev,postgres]"
pytest              # set MINDZEVAL_TEST_POSTGRES_URL to include Postgres
ruff check src tests && ruff format --check src tests && mypy

Work happens on development through pull requests. main is the default branch and production holds released versions.

License

MindzEval is licensed under the Apache License 2.0.

Metadata

Release files for mindzeval 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mindzeval 0.1.0
File Size Uploaded Provenance Actions
mindzeval-0.1.0.tar.gz 115.6 kB With provenance Details for mindzeval-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mindzeval 0.1.0
File Interpreter ABI Platform Actions
mindzeval-0.1.0-py3-none-any.whl Python 3 none any Details for mindzeval-0.1.0-py3-none-any.whl

Total release size: 216.7 kB

Release files / mindzeval-0.1.0.tar.gz

Download URL mindzeval-0.1.0.tar.gz
Size 115.6 kB
Tags Source
SHA-256 checksum
How to use checksums
26703b5856da11b38f99d84d548b08189b39019823a1b6bc54a8c604b795dc61
BLAKE2b-256 checksum
How to use checksums
7bc4ad7c688c0d2cadf50a92fcc3ea6be3cec9d707651d7bb51131ecd526c26e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / mindzeval-0.1.0-py3-none-any.whl

Download URL mindzeval-0.1.0-py3-none-any.whl
Size 101.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
37b64f7ea6c76dca507dd4b2dbb0892e28bf9d1fd2723b075a6936f7f8ce9cc5
BLAKE2b-256 checksum
How to use checksums
db72bb2bcb1e84ee936e64cc2bc5aab7368bc1a275eb733597470cd4dab9da55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for mindzeval 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for mindzeval-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page