deterministic-ai-testing 0.1.0

Deterministic OpenAI-compatible testing infrastructure for AI apps and agents.

deterministic-ai-testing

Playwright for AI agents.

Deterministic OpenAI-compatible testing infrastructure for AI applications, agents, and CI pipelines.

---

The problem

AI applications are extremely difficult to test.

Real LLMs are:

• nondeterministic
• expensive
• slow
• flaky in CI
• hard to reproduce
• difficult to debug

That breaks:

• automated tests
• agent workflows
• CI/CD pipelines
• regression testing
• local development

---

The solution

deterministic-ai-testing is a local OpenAI-compatible mock server that gives you:

✅ deterministic outputs
✅ reproducible AI tests
✅ local/offline development
✅ CI-safe LLM simulation
✅ programmable scenarios
✅ tool-call mocking
✅ streaming simulation
✅ rate-limit/error testing

Without calling real model APIs.

---

Why developers use this

Instead of paying for real inference during tests:

frontend -> real OpenAI API

you can do:

frontend -> deterministic-ai-testing

using only:

base_url="http://localhost:8000/v1"

Everything else stays compatible.

---

Snapshot Testing

Save deterministic AI outputs:

mockllm snapshot save snapshots/hello.json --prompt "hello"

Validate them later:

mockllm snapshot test snapshots/

Example output:

PASS snapshots/hello.json
PASS snapshots/refund.json

This enables:

• deterministic regression testing
• CI-safe AI verification
• stable agent workflows
• reproducible LLM behavior

---

Demo

Start the server

python -m uvicorn app.main:app --reload --port 8000

---

Health check

curl http://localhost:8000/health

Expected:

{"status":"ok","service":"deterministic-ai-testing"}

---

Chat completions

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}]}'

Expected:

{
  "choices": [
    {
      "message": {
        "content": "Hello from MockLLM. This response is deterministic."
      }
    }
  ]
}

---

Tool-call mocking

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"please use tool"}]}'

Expected:

{
  "tool_calls": [
    {
      "function": {
        "name": "search_docs"
      }
    }
  ]
}

---

Error simulation

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"rate limit"}]}'

Expected:

{
  "detail": {
    "error": {
      "message": "Rate limit exceeded by mock scenario."
    }
  }
}

---

Streaming simulation

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","stream":true,"messages":[{"role":"user","content":"hello"}]}'

---

OpenAI SDK compatibility

Works with the official OpenAI SDK.

from openai import OpenAI

client = OpenAI(
    api_key="mock-key",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "hello"}
    ],
)

print(response.choices[0].message.content)

Run:

python3 examples/openai_sdk_example.py

---

60-Second Quickstart

git clone [e-mail]:relentlessreed/deterministic-ai-testing.git

cd deterministic-ai-testing

python3 -m venv .venv

source .venv/bin/activate

pip install -r requirements.txt

python -m uvicorn app.main:app --reload --port 8000

---

Features

OpenAI-compatible endpoints

• /v1/chat/completions
• /v1/completions
• /v1/embeddings

---

Deterministic scenario engine

Supports:

• exact matching
• contains matching
• regex matching

---

YAML scenarios

Edit:

scenarios/default.yaml

Example:

- name: refund request
  match:
    contains: refund
  response:
    content: Refund approved. Your money will be returned in 3-5 business days.

Live edits apply immediately.

No recompilation required.

---

Tool-call simulation

Mock:

• function calls
• tool invocations
• agent actions
• assistant tool responses

Useful for:

• agent testing
• orchestration testing
• frontend AI development

---

Failure simulation

Test:

• rate limits
• retries
• API failures
• fallback logic
• degraded AI behavior

---

Streaming support

Simulate streaming token responses locally.

Useful for:

• chat UIs
• token rendering
• frontend streaming logic
• websocket/SSE behavior

---

Embeddings support

Includes deterministic fake embeddings for:

• RAG testing
• retrieval simulation
• vector pipeline testing
• semantic search development

---

CI-safe AI testing

Run deterministic AI tests in:

• GitHub Actions
• CI/CD pipelines
• local test suites

without paying for inference.

---

Testing

Run all tests:

python -m pytest

---

Docker

Run locally

docker compose up --build

Then test:

curl http://localhost:8000/health

---

Example use cases

• AI app testing
• AI agent testing
• LangChain testing
• RAG testing
• frontend AI UI development
• CI/CD pipelines
• retry/fallback testing
• local AI simulation
• offline AI development
• deterministic regression testing

---

Architecture

application
    ↓
OpenAI SDK
    ↓
deterministic-ai-testing
    ↓
deterministic YAML scenarios

---

Roadmap

Planned features:

• snapshot diffing
• GitHub Actions integration
• installable CLI
• scenario validation
• tool-call assertions
• agent replay
• multi-agent workflows
• hosted team dashboard

---

Product vision

This is not just an OpenAI mock server.

The goal is to become:

deterministic AI testing infrastructure

for the next generation of AI-native software.

or:

Playwright for AI agents.

---

Why this matters

The next major software wave is:

• AI infrastructure
• agent orchestration
• deterministic testing
• AI observability
• developer workflow acceleration

AI software requires a new testing stack.

This project is part of that stack.

---

Contributing

Contributions are welcome.

Especially:

• OpenAI compatibility improvements
• snapshot testing
• LangChain examples
• CLI tooling
• GitHub Actions
• scenario validation
• agent testing workflows

---

License

MIT

Snapshot diffs

When a snapshot fails, MockLLM shows a unified diff:

FAIL snapshots/hello.json
expected exact:
BROKEN EXPECTED OUTPUT

actual:
Hello User - this mock response changed live from YAML.

diff:
--- expected
+++ actual
@@ -1 +1 @@
-BROKEN EXPECTED OUTPUT
+Hello User - this mock response changed live from YAML.

This makes AI output regressions easy to debug in local development and CI pipelines.

---

Scenario validation

MockLLM validates scenarios/default.yaml before using it.

It catches:

• missing match
• missing response or error
• invalid scenario structure
• scenarios that define both response and error

This helps teams fail fast when test scenarios are malformed.