ai-stability 0.1.0

CLI-first LLM stability analyzer for measuring output consistency across repeated prompt runs.

ai-stability

ai-stability is a CLI-first LLM stability analyzer for developers who want to measure output consistency, detect prompt variance, and inspect unstable model behavior locally.

It runs the same prompt multiple times against the same model, compares the responses, computes a simple stability score, and saves a local JSON artifact for replay and debugging.

Why It Exists

LLM outputs often vary even when the prompt, model, and calling code stay the same. That makes it harder to:

• evaluate prompt reliability
• spot regressions during model upgrades
• understand whether output drift is minor wording variance or meaningful behavior change
• build confidence in AI-powered developer tooling

ai-stability is intentionally narrow and local-first:

• one prompt file in
• repeated model calls
• simple, explicit similarity scoring
• readable terminal output
• JSON artifact saved locally for replay and debugging

Features

• CLI-first workflow with no database, dashboard, or hosted backend
• repeated prompt execution against the same model
• explicit pairwise similarity and aggregate stability scoring
• run-by-run output review
• inline reference-vs-run diffing for fast variance inspection
• local JSON artifact saving for debugging and replay
• provider abstraction with OpenAI implemented first

Requirements

• Python 3.11+
• An OpenAI API key in OPENAI_API_KEY

Install

python -m venv .venv
.venv\Scripts\activate
python -m pip install -e .[dev]

Configure

Set your API key in the shell:

$env:OPENAI_API_KEY="your_api_key"

You can copy .env.example for reference, but the CLI reads the key from the environment.

Quick Start

Create a prompt file:

Example prompt.txt:

Explain the tradeoffs between unit tests and integration tests in five bullet points.

Run the analyzer:

ai-stability run prompt.txt --n 5 --provider openai --model gpt-4.1-mini

If you want to invoke it through the module instead of the installed script:

python -m ai_stability run prompt.txt --n 5 --provider openai --model gpt-4.1-mini

Example with a custom JSON output path:

ai-stability run prompt.txt --n 5 --provider openai --model gpt-4.1-mini --out results\sample-run.json

CLI Command

ai-stability run PROMPT_FILE --n 5 --provider openai --model MODEL_NAME

Current options:

• --n: number of repeated runs, minimum 2
• --provider: currently openai
• --model: target model name
• --temperature: sampling temperature, default 1.0
• --out: optional output file or output directory for the JSON artifact

How Scoring Works

The v1 scoring heuristic is intentionally simple and inspectable:

1. normalize whitespace in each output
2. compute pairwise text similarity with Python's difflib.SequenceMatcher
3. average all pairwise similarity scores
4. convert the average to a 0-100 stability score

Stability labels:

• 80-100: High stability
• 50-79: Medium stability
• 0-49: Low stability

What the CLI Prints

• summary first
• average and pairwise similarity
• final stability score and label
• each run output
• a simple reference-vs-run diff for variation review

JSON Artifact

By default, results are written to results/ai-stability-YYYYMMDD-HHMMSS.json.

The JSON artifact includes:

• prompt metadata
• provider and model
• all collected outputs
• pairwise similarities
• stability score and label
• human-readable diffs

Example Workflow

ai-stability run prompt.txt --n 5 --provider openai --model gpt-4.1-mini

Use this when you want to compare how stable a model is for a fixed prompt before shipping a prompt change, swapping models, or debugging flaky output behavior.

Run Tests

python -m pytest

Repository Structure

src/ai_stability/
  cli.py
  runner.py
  scoring.py
  diffing.py
  output.py
  storage.py
  providers/
    base.py
    openai_provider.py
tests/
  test_scoring.py
  test_runner.py

Files to Review First

• src/ai_stability/cli.py
• src/ai_stability/runner.py
• src/ai_stability/scoring.py
• src/ai_stability/providers/openai_provider.py

Roadmap Notes

• V1 runs requests sequentially on purpose.
• Only OpenAI is implemented, but the provider boundary is small and ready for Anthropic later.
• The scoring heuristic is intentionally simple and inspectable rather than statistically sophisticated.