lightassay 0.3.3

File-based orchestrator for structured evaluation of applied LLM workflows: humans declare intent, LLMs reason about quality, code runs and records raw facts

Vibe-testing LLM workflows for vibe coders

lightassay is AI-native evaluation for real LLM workflows: describe what worries you, let an agent build the workbook, run the actual workflow, and review semantic analysis. No fake coded quality metrics.

Install

pip install lightassay

Setup

Requires an installed and authenticated Codex CLI or Claude CLI.

lightassay init

Quick Start

lightassay quickstart \
  --message "Check the main workflow behavior, edge cases, and obvious failure modes." \
  --target "python:myapp.pipeline.run"

Use Cases

• Create a first eval suite from plain-language intent.
• Rerun the same workbook after app fixes.
• Compare target models from one workbook.
• Connect Python, JavaScript, TypeScript, HTTP, or command-based workflows.
• Keep evaluation state in local workbook and JSON artifacts.

Commands

• lightassay quickstart - create a workbook, run the target workflow, and write analysis.
• lightassay continue - continue an existing workbook with new instructions.
• lightassay rerun - rerun the same prepared workbook after app changes.
• lightassay compare-models - run the workbook model matrix and compare results.
• lightassay adapter-template - generate a thin target adapter template.
• lightassay adapter-check - validate a target adapter before running evals.
• lightassay agent-guide - print installed-package guidance for coding agents.

Why Lightassay

• AI-first: built for agent-driven eval creation and analysis.
• Agent-operated: uses your authenticated Codex CLI or Claude CLI subscription for preparation, analysis, and compare instead of a separate analysis API bill.
• File-based: workbook markdown plus JSON artifacts.
• Real workflow boundary: tests your target workflow, not isolated prompt snippets.
• No fake coded quality metrics.

Docs

For installed-package guidance:

lightassay agent-guide

Repository docs:

• Quickstart
• Agent operational guide

License

MIT - see LICENSE.