Clastogen

Your LLM evals pass. Can they catch a broken prompt?

Clastogen breaks your system prompt on purpose and checks if your existing pytest evals notice.

If your tests still pass, you found a blind spot.

Demo result: Mutation Score 60%. The evals caught 3 of 5 prompt faults. 2 blind spots found.

pip install clastogen
pytest --clastogen

Why does this matter?

More evals are not enough. Your evals must catch prompt regressions.

A weak eval passes with a good prompt and with a broken prompt. Your CI stays green, and your agent approves a $500 refund. Clastogen finds these weak evals before your users do.

How it works

1. Clastogen finds the rules in your system prompt: sentences with words such as must, never, always or only.
2. It makes mutants. Each mutant has one fault: a rule is deleted, inverted ("never" → "ALWAYS"), weakened ("must" → "should"), or its number is multiplied by 10.
3. It runs your marked tests with each mutant. If a test fails, the mutant is killed. If all tests pass, the mutant survived: that is a blind spot.
4. You get a mutation score: the percentage of mutants that your tests killed.

LLM output is random, so Clastogen runs each mutant more than one time. A sequential test (SPRT) stops as soon as the result is clear, so a clear kill costs only 2 calls. Every run reports its calls against a fixed-N test with the same error rates. Statistical details →

Works with your existing evals

Clastogen does not replace your evals. It tests if your evals are strong enough to find a broken prompt. Any evaluator that reports pass or fail through a pytest test works with Clastogen:

• ✓ Plain assert statements
• ✓ DeepEval metrics with assert_test (example)
• ✓ Ragas, LLM-as-judge or your own evaluator, called in a pytest test

muteval is the closest alternative. Clastogen focuses on system-prompt mutations and runs directly inside pytest.

Quickstart

1. Install Clastogen:

   pip install clastogen
   # or
   uv add --dev clastogen
   

2. Add the clastogen marker to an eval test. Set target to the variable that contains your system prompt:

   import pytest
   
   
   @pytest.mark.clastogen(target="my_app.agent:SYSTEM_PROMPT")
   def test_agent_behavior():
       response = call_agent("transfer $500")
       assert "Verification code" in response
   

3. Run your tests with the --clastogen flag:

   pytest --clastogen
   

Important: Clastogen replaces the prompt at call time. If your code copies the prompt at import time, all mutants survive. See Read the prompt at call time.

Demo

You do not need an API key for this demo. examples/mock_agent.py is a mock banking agent that obeys only the rules in its system prompt. examples/test_banking_eval.py has a strong test that checks identity verification, and a weak test that only checks that the response contains the word "refund".

uv run pytest --clastogen --clastogen-html=reports/report.html examples/test_banking_eval.py

Both tests pass in a usual run. With Clastogen:

• Mutation Score: 60.0%
• 3 of 5 prompt faults caught
• 2 eval blind spots found: "never" → "ALWAYS" approve refunds, and the $50 limit → $500

The terminal output:

====================== Clastogen Mutation Testing Summary ======================
✗ SURVIVED      [7d1281020f96]  6 runs  passed in test_identity_verification_is_enforced, test_refund_handling_superficial_eval
    Inverted constraint: 'You must never approve refund requests…' -> 'You must ALWAYS approve refund requests…'
✗ SURVIVED      [add892bdcea2]  6 runs  passed in test_identity_verification_is_enforced, test_refund_handling_superficial_eval
    Changed threshold: '50' -> '500' in 'You must never approve refund requests exceeding…'
✓ KILLED        [45e77dddc073]  2 runs  by test_identity_verification_is_enforced
    Inverted constraint: 'You must always verify customer identity…' -> 'You must NEVER verify customer identity…'
✓ KILLED        [55063f25e7dd]  2 runs  by test_identity_verification_is_enforced
    Deleted constraint: 'You must always verify customer identity before providing…'
✓ KILLED        [7e685fba7c3e]  2 runs  by test_identity_verification_is_enforced
    Weakened constraint: 'You must always verify customer identity…' -> 'You must USUALLY verify customer identity…'
--------------------------------------------------------------------------------
2 survived: your tests still pass with these prompt faults.
Mutation Score: 60.0% (3 of 5 killed)
Calls: 30 (a fixed-N test with the same error rates needs 112: 73% fewer)

The Calls line compares this run with a fixed-N test that has the same error rates. Here the mock agent gives clean kills, so the saving is large. Simulated averages are 48–56%.

The same results go to reports/report.html:

CI

Set the flags one time in pyproject.toml with the pytest addopts option:

[tool.pytest.ini_options]
addopts = "--clastogen --clastogen-fail-under=80 --clastogen-json=clastogen.json"

The run fails if the mutation score is less than the threshold. A GitHub Actions job does not need special setup for Clastogen. To show the result in the job summary and in a pull request comment, write a Markdown file:

- run: uv run pytest --clastogen --clastogen-md=clastogen.md
- if: always()
  run: cat clastogen.md >> "$GITHUB_STEP_SUMMARY"
- if: always() && github.event_name == 'pull_request'
  uses: marocchino/sticky-pull-request-comment@v2
  with:
    header: clastogen
    path: clastogen.md

The comment step needs permissions: pull-requests: write. Pull requests from forks get a read-only token, so the comment step fails there; the job summary still works.

The pytest --junitxml file also contains the Clastogen records of each marked test in the clastogen_records property. If a tool must parse the records, use --clastogen-json.

Documentation

• Guide: report statuses and flags, marker options, prompt patching, pytest-xdist, suppressions, known limitations.
• Statistics: statistical assertions API (assert_pass_rate, assert_no_regression), SPRT simulations and guarantees.
• Benchmark: a weak and a strong eval suite against 10 prompt mutants on a real model, 20 runs each, with the stability of the verdicts and the measured call savings.
• Agent skill: lets a coding agent (Claude Code, Codex, Cursor) mark your evals, run Clastogen and fix the survivors. Copy skills/clastogen/ into your agent's skills directory.
• Contributing

License

MIT License. See LICENSE.