eval-case-curator 0.1.0

Local CLI and desktop app for turning AI failure notes into reusable eval cases

Eval Case Curator

Eval Case Curator turns messy AI failure notes, support escalations, prompt review comments, and QA findings into reusable eval cases.

It runs locally, redacts common contact/credential patterns, and exports Markdown, JSON, JSONL, or HTML reports that can be dropped into prompt regression suites or review notes.

Install

pip install eval-case-curator

CLI

eval-case-curator examples/failure-notes.txt
eval-case-curator examples/failure-notes.txt --format jsonl --output evals.jsonl
eval-case-curator examples/failure-notes.txt --fail-under 3

Desktop App

eval-case-curator-gui

Paste notes, curate cases, copy JSONL, or save a Markdown report.

What It Extracts

• Candidate eval input from Input:, Prompt:, User:, or fallback context.
• Expected behavior from Expected:, Should:, Desired:, or failure wording.
• Priority from safety, privacy, financial, legal, hallucination, format, and missing-answer clues.
• Tags such as privacy, format, retrieval, instruction-following, tool-use, and tone.
• Source line references and risk notes for review traceability.

Example

eval-case-curator examples/failure-notes.txt --format jsonl

{"id":"eval-...","input":"Summarize the renewal risk...","expected":"Mention renewal risk without exposing direct contact details.","metadata":{"priority":"high","tags":["privacy"]}}

Development

PYTHONPATH=src python3 -m unittest discover -s tests
python3 -m build --no-isolation
python3 -m twine check dist/*

License

MIT