MindzEval
MindzEval proves your SQL agent is accurate and safe before it reaches users, and keeps proving it on every change.
Status: release
0.1.0. New here? Start with getting started. Feedback and bug reports are welcome in the issues.
What it does
- Grades the data, not the SQL. The agent's results are compared with the results of a reference query, so correct SQL written a different way still passes.
- Checks the answer key first. Reference queries are validated against the live database (empty, drift, unstable, NULL groups, orphan-inflated) before any agent is graded.
- Scores behaviour both ways. Asking for clarification or declining is rewarded when it's right and penalised when it isn't.
- Separates display limits from logic errors. The agent's SQL is re-run without its display
LIMIT, so a UI row cap isn't mistaken for a wrong query. - Checks safety. Write attempts, forbidden tables and PII in results or answers.
- Works with any agent. LangChain, LangGraph, CrewAI, LlamaIndex, the OpenAI Agents SDK, plain Python, or any HTTP service.
- Compares models and prompts. Matrix runs evaluate every combination side by side.
- Read-only by design. Every query MindzEval runs is parsed and refused unless it is a single read, then executed in a read-only transaction with a timeout.
Installation
MindzEval requires Python 3.11 or newer:
# SQLite only
pip install mindzeval
# With PostgreSQL support
pip install "mindzeval[postgres]"
Quickstart
mindzeval quickstart demo
cd demo
mindzeval run
The example agent fails some cases on purpose so you can see what each verdict means.
Your own agent
mindzeval.yaml:
project: shop-agent
agent:
type: python
target: my_project.eval_adapter:answer # or type: http with a URL
database:
url: postgresql://reader:${DB_PASSWORD}@db.internal/shop
datasets: cases/*.yaml
cases/customers.yaml:
- id: customers_in_germany
question: Which customers are in Germany?
reference_sql: SELECT id FROM customers WHERE country = 'DE'
key: id
- id: vague_request
question: Show me the data
expect: {behavior: clarify}
mindzeval lint # check config and cases (no connections, no secrets)
mindzeval validate # check reference queries against the database
mindzeval run # evaluate; exits 1 if thresholds aren't met
Documentation
- Getting started: install, try the example, connect your agent
- Integration examples: runnable examples per framework, starting with LangGraph
- Connecting your agent: Python, LangGraph, LangChain, CrewAI, HTTP, saved outputs, model comparison
- Case format
- Config file
- Agent result record
- Reading results: verdicts, numbers, failure causes, exit codes
- Custom scorers
Development
git clone https://github.com/MindzKonnected/mindzeval.git && cd mindzeval
python -m venv .venv && .venv\Scripts\activate # macOS/Linux: source .venv/bin/activate
pip install -e ".[dev,postgres]"
pytest # set MINDZEVAL_TEST_POSTGRES_URL to include Postgres
ruff check src tests && ruff format --check src tests && mypy
Work happens on development through pull requests. main is the default branch and production holds released versions.
License
MindzEval is licensed under the Apache License 2.0.
Metadata
Release files for mindzeval 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mindzeval-0.1.0.tar.gz | 115.6 kB | With provenance | Details for mindzeval-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mindzeval-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for mindzeval-0.1.0-py3-none-any.whl |
Total release size: 216.7 kB
Release files / mindzeval-0.1.0.tar.gz
| Download URL | mindzeval-0.1.0.tar.gz |
|---|---|
| Size | 115.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
26703b5856da11b38f99d84d548b08189b39019823a1b6bc54a8c604b795dc61
|
|
BLAKE2b-256 checksum How to use checksums |
7bc4ad7c688c0d2cadf50a92fcc3ea6be3cec9d707651d7bb51131ecd526c26e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logRelease files / mindzeval-0.1.0-py3-none-any.whl
| Download URL | mindzeval-0.1.0-py3-none-any.whl |
|---|---|
| Size | 101.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
37b64f7ea6c76dca507dd4b2dbb0892e28bf9d1fd2723b075a6936f7f8ce9cc5
|
|
BLAKE2b-256 checksum How to use checksums |
db72bb2bcb1e84ee936e64cc2bc5aab7368bc1a275eb733597470cd4dab9da55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logSecurity for mindzeval 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for mindzeval-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.