shouldpy

"Write your tests the way you always have. Let AI run one final check."

What is this?

A ~200-line decorator that wraps any test function with a natural-language assertion. After the function runs, an LLM inspects all logs, print output, and return values to check whether the condition holds:

• PASS — passes silently
• FAIL: reason — raises AssertionError

Does not replace assert. Does not force a return. Does not restructure your test cases. Just one decorator that lets an LLM double-check your work.

Install

pip install shouldpy

Quick start

import should

should.use("openai/gpt-4o-mini", api_key="sk-...")

@should("The log should clearly indicate the order was created successfully")
def test_create_order():
    ...  # your existing test code, unchanged
    logging.info("Order created successfully")
    assert resp.status_code == 200

Run pytest →

• Your existing assert checks the status code
• AI additionally checks whether logs/output/return value satisfy the natural-language condition

Both must pass.

Supported models

should.use() accepts any model URI supported by chak:

should.use("openai/gpt-4o-mini",        api_key="...")
should.use("anthropic/claude-3-5-sonnet", api_key="...")
should.use("deepseek/deepseek-chat",    api_key="...")
should.use("ollama/llama3",             base_url="http://localhost:11434")

Async support

@should("Async minor registration must be rejected")
async def test_async_register_minor():
    result = await async_register_user("Alice", 15)
    return result

Works out of the box — async functions get a fully async AI call.

Notes

1. Bring your own API key. shouldpy calls the LLM you configure; it never manages credentials itself.
2. Non-deterministic. Best suited for exploratory testing, scripting, local spot-checks, and pre-review sanity passes. Do not use as a primary assertion in high-frequency CI pipelines.
3. Every call hits the model — slow, costs tokens, and may expose data. For sensitive workloads, use a local model (e.g. Ollama).

Feedback

https://github.com/zhixiangxue/should-ai

Issues, PRs, and stars are welcome.

License

MIT © 2025 zx