Clastogen

Your LLM evals pass. Can they catch a broken prompt?

Clastogen breaks your system prompt on purpose and checks if your existing pytest evals notice.

If your tests still pass, you found a blind spot.

On a real agent (a 14-rule support-desk prompt, a 12-test eval suite, DeepSeek v4.1 flash), the evals killed only 27.1% of the mutants. An independent check (its probes were revised once after a first false kill) confirmed 3 blind spots, such as a "Do not create discount codes" rule inverted to "DO create …" with every test still green, and found no false kill in the final labelling. Case study →

Demo result: Mutation Score 60%. The evals caught 3 of 5 prompt faults. 2 blind spots found.

pip install clastogen
pytest --clastogen

Why does this matter?

More evals are not enough. Your evals must catch prompt regressions.

A weak eval passes with a good prompt and with a broken prompt. Your CI stays green, and your agent approves a $500 refund. Clastogen finds these weak evals before your users do.

How it works

1. Clastogen finds the rules in your system prompt: sentences with a word such as must, never, always, only, if, unless, escalate or refuse, or with a number. Role sentences ("You are …") are skipped.
2. It makes mutants. Each mutant has one fault: a rule is deleted, inverted ("never" → "ALWAYS"), weakened ("must" → "should"), its number is multiplied by 10, its "unless …" clause is dropped, its comparator is flipped ("at most" → "AT LEAST"), its scope is narrowed ("all" → "SOME"), or a whole section or <example> is deleted. Every rule gets one mutant before any rule gets a second, so a small max_mutants still spreads over the prompt.
3. It runs your marked tests with each mutant. If a test fails, the mutant is killed. If all tests pass, the mutant survived: that is a blind spot.
4. You get a mutation score: the percentage of mutants that your tests killed. Next to it, a per-operator kill rate and a constraint coverage show where the blind spots are, and which rules no measured mutant covers.

LLM output is random, so Clastogen runs each mutant more than one time. A sequential test (SPRT) stops as soon as the result is clear, so a clear kill costs only 2 calls. Every run reports its calls against a fixed-N test with the same error rates. Statistical details →

A survivor is not always a blind spot: the model can keep its behavior without a rule. Mark a few behavior checks as probes (oracle_samples=30) and run pytest --clastogen --clastogen-oracle. Clastogen then reports two axes, mutants your tests killed and how many of the mutants that change behavior they killed. Probes →

Works with your existing evals

Clastogen does not replace your evals. It tests if your evals are strong enough to find a broken prompt. Any evaluator that reports pass or fail through a pytest test works with Clastogen:

• ✓ Plain assert statements
• ✓ DeepEval metrics with assert_test (example)
• ✓ Ragas, LLM-as-judge or your own evaluator, called in a pytest test

Quickstart

1. Install Clastogen:

   pip install clastogen
   # or
   uv add --dev clastogen
   

2. Add the clastogen marker to an eval test. Set target to the variable that contains your system prompt:

   import pytest
   
   
   @pytest.mark.clastogen(target="my_app.agent:SYSTEM_PROMPT")
   def test_agent_behavior():
       response = call_agent("transfer $500")
       assert "Verification code" in response
   

3. Run your tests with the --clastogen flag:

   pytest --clastogen
   

Important: Clastogen replaces the prompt at call time. If your code copies the prompt at import time, all mutants survive. See Read the prompt at call time.

Demo

This demo is a mock agent, so it needs no API key. For a real model, see the case study below. examples/mock_agent.py is a mock banking agent that obeys only the rules in its system prompt. examples/test_banking_eval.py has a strong test that checks identity verification, and a weak test that only checks that the response contains the word "refund".

uv run pytest --clastogen --clastogen-html=reports/report.html examples/test_banking_eval.py

Both tests pass in a usual run. With Clastogen:

• Mutation Score: 60.0%
• 3 of 5 prompt faults caught
• 2 eval blind spots found: "never" → "ALWAYS" approve refunds, and the $50 limit → $500

The terminal output:

====================== Clastogen Mutation Testing Summary ======================
✗ SURVIVED      [7d1281020f96]  6 runs  passed in test_identity_verification_is_enforced, test_refund_handling_superficial_eval
    Inverted constraint: 'You must never approve refund requests…' -> 'You must ALWAYS approve refund requests…'
✗ SURVIVED      [add892bdcea2]  6 runs  passed in test_identity_verification_is_enforced, test_refund_handling_superficial_eval
    Changed threshold: '50' -> '500' in 'You must never approve refund requests exceeding…'
✓ KILLED        [45e77dddc073]  2 runs  by test_identity_verification_is_enforced
    Inverted constraint: 'You must always verify customer identity…' -> 'You must NEVER verify customer identity…'
✓ KILLED        [55063f25e7dd]  2 runs  by test_identity_verification_is_enforced
    Deleted constraint: 'You must always verify customer identity before providing…'
✓ KILLED        [7e685fba7c3e]  2 runs  by test_identity_verification_is_enforced
    Weakened constraint: 'You must always verify customer identity…' -> 'You must USUALLY verify customer identity…'
--------------------------------------------------------------------------------
2 survived: your tests still pass with these prompt faults.
Mutation Score: 60.0% (3 of 5 killed)
Calls: 30 (a fixed-N test with the same error rates needs 112: 73% fewer)
By operator: change_threshold 0/1 (0%), delete_constraint 1/1 (100%), invert_negation 1/2 (50%), weaken_modal 1/1 (100%)
Constraint Coverage: 1 of 2 constraints killed (50.0%); 1 survived or inconclusive, 0 untested

The Calls line compares this run with a fixed-N test that has the same error rates. Here the mock agent gives clean kills, so the saving is large. Simulated averages are 48–56%.

The same results go to reports/report.html:

Real models

The support-desk case study tests a 14-rule e-commerce support agent on DeepSeek with a 12-test suite, and checks every verdict against an independent set of behavior probes. The suite scored 27.1% (5 runs). None of the 9 kills was false, 3 survivors were confirmed blind spots (no test for the language, refund-count or discount rules), and 11 survivors were mutants that do not change this model's behavior. 4 mutants were too close to the threshold to call, and 7 mutants (of 3 rules) have no probe.

CI

Set the flags one time in pyproject.toml with the pytest addopts option:

[tool.pytest.ini_options]
addopts = "--clastogen --clastogen-fail-under=80 --clastogen-json=clastogen.json"

The run fails if the mutation score is less than the threshold. A GitHub Actions job does not need special setup for Clastogen. To show the result in the job summary and in a pull request comment, write a Markdown file:

- run: uv run pytest --clastogen --clastogen-md=clastogen.md
- if: always()
  run: cat clastogen.md >> "$GITHUB_STEP_SUMMARY"
- if: always() && github.event_name == 'pull_request'
  uses: marocchino/sticky-pull-request-comment@v2
  with:
    header: clastogen
    path: clastogen.md

The comment step needs permissions: pull-requests: write. Pull requests from forks get a read-only token, so the comment step fails there; the job summary still works.

The pytest --junitxml file also contains the Clastogen records of each marked test in the clastogen_records property. If a tool must parse the records, use --clastogen-json.

Documentation

• Guide: report statuses and flags, marker options, prompt patching, pytest-xdist, suppressions, known limitations.
• Statistics: statistical assertions API (assert_pass_rate, assert_no_regression), SPRT simulations and guarantees.
• Case study: a realistic support agent, a 12-test suite and an independent oracle that tells blind spots from equivalent mutants.
• Agent skill: lets a coding agent (Claude Code, Codex, Cursor) mark your evals, run Clastogen and fix the survivors. Copy skills/clastogen/ into your agent's skills directory.
• Contributing

License

MIT License. See LICENSE.