Clastogen

Your LLM evals pass. Do they find a broken prompt?

Clastogen is a pytest plugin. It does mutation testing on the system prompts of LLM apps and AI agents.

Clastogen puts small faults in your prompt. It removes a rule, changes "never" to "always", or multiplies a limit by 10. Then it runs your tests again. If your tests continue to pass, they cannot find that fault. Clastogen uses a Sequential Probability Ratio Test (SPRT). The SPRT stops when the result is clear, so you use fewer API calls.

Quickstart

1. Install Clastogen:

   pip install clastogen
   # or
   uv add --dev clastogen
   

2. Add the clastogen marker to an eval test. Set target to the variable that contains your system prompt:

   import pytest
   
   
   @pytest.mark.clastogen(target="my_app.agent:SYSTEM_PROMPT")
   def test_agent_behavior():
       response = call_agent("transfer $500")
       assert "Verification code" in response
   

3. Run your tests with the --clastogen flag:

   pytest --clastogen
   

After the test run, Clastogen shows a mutation score. A low score tells you that your tests do not find faults in your prompt.

For local development, see CONTRIBUTING.md.

Demo

You do not need an API key for this demo. examples/mock_agent.py is a mock banking agent. The agent obeys only the rules that its system prompt contains. examples/test_banking_eval.py has two evals:

• A strong test. It makes sure that the agent asks for identity verification.
• A weak test. It only makes sure that the response is not empty and contains the word "refund".

uv run pytest --clastogen --clastogen-html=reports/report.html examples/test_banking_eval.py

Clastogen writes the summary to the terminal:

It also writes the same results to reports/report.html:

The two tests pass in a usual run. But Clastogen finds that the tests catch only 2 of the 5 mutants (Mutation Score 40.0%):

• The strong test kills the two identity mutants. Each mutant needs only 2 runs.
• The weak test continues to pass when Clastogen removes the refund rule. It also passes when the rule changes to "ALWAYS approve" and when the limit changes from $50 to $500. These three mutants survive. They show a blind spot in your evals.

How to read the report

• Statuses:
  • KILLED: a test failed with the mutant.
  • SURVIVED: the tests continued to pass with the mutant.
  • INCONCLUSIVE: the SPRT got to max_steps before a decision.
  • ERROR: the evaluation itself failed. The JSON error field gives the cause.
  • SKIPPED: the test called pytest.skip during a trial.
  • SUPPRESSED: you suppressed the mutant. See Suppress mutants.
• Mutation Score = KILLED / (KILLED + SURVIVED + INCONCLUSIVE). The score does not include ERROR, SKIPPED and SUPPRESSED mutants. If more than one test examines a mutant, the strongest result is the result for that mutant. The order is KILLED > SURVIVED > INCONCLUSIVE > ERROR > SKIPPED > SUPPRESSED.
• Baselines: Before mutation, Clastogen runs each marked test 9 more times without a mutant (10 runs in total). It calculates p0 with the Laplace estimate (s + 1) / (n + 2). If the raw pass rate of a test is less than 80%, the test is flaky and Clastogen does not mutate it. The terminal summary shows the pass counts on one "Baselines" line. Use -v to show p0 for each test.
• Flags:
  • --clastogen-fail-under MIN_SCORE: the run fails if the score is less than MIN_SCORE.
  • --clastogen-json PATH: writes the results to a JSON file. The file contains mutation_score, total_mutants, counts (all six statuses), results (one merged record for each mutant), executions (the record of each test for each mutant, which the HTML kill matrix uses) and baselines. For mutants that the SPRT did not run (ERROR, SKIPPED, SUPPRESSED), sample_count and llr are null.
  • --clastogen-html PATH: writes the same data to one HTML file.

How Clastogen compares

Most LLM eval tools examine the output of your model. Clastogen examines your tests. Thus, you can use Clastogen together with these tools.

Clastogen runs on each pytest test that has the clastogen marker. This includes tests that use metrics from other eval libraries.

Usage

1. Pytest plugin

Run your tests with mutation testing:

# Run all tests with mutation testing
pytest --clastogen

# Run one test file
pytest --clastogen tests/test_agent.py

# Show a log for each step
pytest --clastogen -s --log-cli-level=INFO

Marker options

@pytest.mark.clastogen(
    target="my_app.agent:SYSTEM_PROMPT",  # Required: 'module:VAR' or 'module:Class.ATTR'
    max_mutants=5,  # Maximum number of mutants (default: 5)
    delta=0.30,  # Minimum pass-rate decrease to find (default: 0.30)
    p0=0.90,  # Known baseline pass rate (default: measured in 10 baseline runs; if you set it, Clastogen does not do these runs)
    max_steps=20,  # Maximum number of runs for each mutant (default: 20)
)

Read the prompt at call time

For each trial, Clastogen replaces the module attribute (or class attribute) with the mutant. After the trial, it puts back the original value. It also replaces module-level aliases that have the same name and refer to the same string object. An example is from my_app.agent import SYSTEM_PROMPT at the top of a different module.

Thus, your code must read the prompt when it calls the model. Clastogen does not change copies that your code made before the trial. Mutants of these copies always show SURVIVED.

For each trial, Clastogen removes and builds again the function-scoped fixtures. Thus, a fixture such as bot = {"system": agent.PROMPT} gets the mutant. Clastogen builds module-scoped and session-scoped fixtures only one time, so they keep the original prompt. Values that your code captures at import time also keep the original prompt. Mutants of these values also show SURVIVED.

Incorrect: the code captures the prompt one time at import time. It does not read the changed attribute.

# my_app/agent.py
SYSTEM_PROMPT = "You must never approve refunds over $50."
# A copy made at import. F-strings, default args and clients made at import have the same problem.
CONFIG = {"system": SYSTEM_PROMPT}


def ask(question: str) -> str:
    return client.chat(system=CONFIG["system"], user=question)

Correct: the code reads the module attribute at each call. A from my_app.agent import SYSTEM_PROMPT in the function body also works.

# my_app/agent.py
SYSTEM_PROMPT = "You must never approve refunds over $50."


def ask(question: str) -> str:
    return client.chat(system=SYSTEM_PROMPT, user=question)

Parallel runs (pytest-xdist)

You can use pytest --clastogen -n 4. Each worker sends its records with the test report. The controller merges the records into one summary. Each process measures its own baselines and keeps its own list of killed mutants. Thus, a worker can run a mutant again after a different worker killed it. Parallel runs do more work in total, but they take less time.

CI

Set the flags one time in pyproject.toml with the pytest addopts option:

[tool.pytest.ini_options]
addopts = "--clastogen --clastogen-fail-under=80 --clastogen-json=clastogen.json"

A GitHub Actions job does not need special setup for Clastogen:

- run: pip install clastogen
- run: pytest --junitxml=junit.xml

The run fails if the mutation score is less than the threshold. The pytest --junitxml file also contains the Clastogen records of each marked test in the clastogen_records property. If a tool must parse the records, use --clastogen-json.

2. Statistical assertions (Python API)

One assert on LLM output is flaky. Use these functions to get statistical guarantees:

• assert_pass_rate calls a callable that has no arguments and returns a bool. It uses Wald's SPRT and stops when the evidence is sufficient. It calls the callable max_samples times at most. If the pass rate is less than min_rate, it raises AssertionError. See Pass-rate guarantees.
• assert_no_regression calls a baseline callable and a candidate callable. By default, it uses a paired McNemar test. With paired=False, it uses a Fisher exact test. It raises AssertionError only if the decrease is statistically significant.
• evaluate_pass_rate and evaluate_regression return the same result objects, but they do not assert.
• compute_wilson_interval is also available.

The example below is examples/test_stats_usage.py. It uses fake evaluators with a seed. Thus, you can run it with pytest examples/test_stats_usage.py and without an API key:

"""Statistical assertions on seeded fake evaluators: zero API keys, deterministic."""

import random
from collections.abc import Callable

import pytest

from clastogen import assert_no_regression, assert_pass_rate


def fake_llm(pass_rate: float, seed: int) -> Callable[[], bool]:
    rng = random.Random(seed)
    return lambda: rng.random() < pass_rate


def test_order_intent_pass_rate() -> None:
    result = assert_pass_rate(fake_llm(0.97, seed=1), min_rate=0.90, description="order intent")
    assert result.observed_rate >= 0.90


def test_prompt_upgrade_does_not_regress() -> None:
    assert_no_regression(fake_llm(0.90, seed=2), fake_llm(0.90, seed=3), description="prompt v1 -> v2")


def test_broken_prompt_is_detected_as_regression() -> None:
    with pytest.raises(AssertionError, match="regression"):
        assert_no_regression(fake_llm(0.95, seed=4), fake_llm(0.40, seed=5), description="prompt v1 -> v3")

Suppress mutants

Sometimes a mutant survives because the foundation model obeys the rule from its training. This is an equivalent mutant. Suppress equivalent mutants, so that they do not change your score. Clastogen shows suppressed mutants as SUPPRESSED, and the Mutation Score does not include them. The mutant ID is the 12-character hash in the summary. An example is 7d1281020f96, the inverted refund rule in the Demo.

Put the IDs in .clastogen/suppressions.toml in your pytest rootdir. Each entry must have a reason:

[[suppressions]]
mutant_id = "7d1281020f96"
reason = "Model refuses large refunds regardless of the prompt"

If the file is not correct, a --clastogen run stops with a usage error. Runs without --clastogen do not read the file.

SPRT performance and limits

The SPRT uses Wald's sequential boundaries to set the number of samples. For a severe defect with a measured 10/10 baseline, the SPRT stops after only 2 calls. With an explicit $p_0 = 0.90$, it stops after 3 calls. For results near the boundary, the SPRT stops at $N_{\max}$.

These results come from a Monte Carlo simulation (PYTHONPATH=src uv run python scripts/simulate_sprt.py, 10,000 trials for each row, seed 42):

Scenario 1: Standard Evaluation ($p_0 = 0.90, p_1 = 0.60, N_{\max} = 20$)

Scenario 2: Flaky Baseline ($p_0 = 0.75, p_1 = 0.50, N_{\max} = 25$)

Scenario 3: Full Plugin Flow (10-run Laplace baseline, $\delta = 0.30$, $N_{\max} = 20$)

The plugin runs the test without a mutant 10 times. It calculates $p_0 = (s + 1) / (n + 2)$ (the Laplace rule of succession). If the raw pass rate is less than 80%, the plugin rejects the baseline. Then it runs the SPRT on each mutant with $p_1 = p_0 - 0.30$. The rates below include only the accepted baselines.

Indifference zone: Sometimes the mutant pass rate is near $(p_0 + p_1) / 2$. In this zone, a sequential test cannot make a decision with a finite number of samples. Clastogen stops at $N_{\max}$ and shows the result as INCONCLUSIVE. It does not guess.

Pass-rate guarantees

assert_pass_rate tests $H_0$: rate $=$ min_rate against $H_1$: rate $=$ min_rate - tolerance. Both error rates are 1 - confidence.

• If the true rate is min_rate or more, the assertion passes with a probability of approximately confidence.
• If the true rate is min_rate - tolerance or less, the assertion fails with approximately the same probability.
• If the true rate is between these two values (the indifference zone), the assertion can pass or fail.

If the SPRT uses all max_samples before a decision, the sign of the log-likelihood ratio gives the decision. result.decided tells you if the SPRT stopped by itself. A smaller tolerance gives a more precise result, but it needs more samples.

Scenario 4 of the simulation script, with the default values (min_rate=0.90, tolerance=0.20, confidence=0.95, max_samples=50):

Known limitations

1. Mutation operators: Clastogen finds rules by their words. A sentence is a candidate only if it contains a keyword such as must, never, always, avoid, only, may not or require. Clastogen does not mutate rules without one of these keywords, for example "Escalate to a human if …". Clastogen applies three operators to each rule: delete_constraint, invert_negation and change_threshold (the first number × 10). Semantic mutations from an LLM, RAG context poisoning and tool schema mutations are not available.
2. Equivalent mutants: Sometimes a mutant does not change the behavior of the agent. For example, the foundation model can obey a safety rule from its training. Clastogen does not prove semantic equivalence. Use suppressions (.clastogen/suppressions.toml) for these mutants.
3. Effect size: The SPRT looks for an absolute decrease of delta (default 0.30) from a 10-run Laplace baseline. Thus, the maximum p0 is 11/12 ≈ 0.917. A small regression such as 0.95 → 0.88 needs approximately 60-70 runs for each mutant. With max_steps=20, the result is INCONCLUSIVE or SURVIVED. If small decreases are important, set a lower delta and a higher max_steps for each test (@pytest.mark.clastogen(delta=0.10, max_steps=100)). This increases the API cost.
4. Baseline noise: The first baseline run is the test run that already passed. Also, p0 is a point estimate. Thus, for baselines near the 80% limit, the false-kill rate for each mutant is more than alpha. These are the rates for an unchanged mutant (delta=0.30, max_steps=20, accepted baselines only): a true baseline of 0.90 gives 2.2%, 0.85 gives 5.0%, 0.80 gives 8.6% and 0.75 gives 13.2%. A higher limit does not fix this problem. With a 90% limit, 0.85 still gives 7.0%. Make the test more stable, or set p0 explicitly.
5. Differential execution: After a test kills a mutant, other tests do not run that mutant again. But Clastogen does not make a static dependency graph before execution.

Commands summary

License

MIT License. See LICENSE.