Sequa 📼

Deterministic testing for AI applications.

Record once. Replay forever.

Stop paying for every AI test run.

Every time your AI application runs during testing, it probably:

• 💸 Calls the LLM again
• 🐢 Slows down your CI pipeline
• 🎲 Produces slightly different outputs
• 🌐 Depends on internet connectivity

Sequa records a real AI execution once and replays it locally during future test runs.

The result

• ⚡ Millisecond replay
• 💰 Zero replay API costs
• 🧪 Deterministic testing
• 💻 Works offline

Before

from langchain_groq import ChatGroq

model = ChatGroq(model_name="llama-3.1-8b-instant")

response = model.invoke(
    "Write a 3-word slogan for gravity."
)

# ⏱️ 2.3 seconds
# 🌐 Live API Call

After

from langchain_groq import ChatGroq
from sequa import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke(
        "Write a 3-word slogan for gravity."
    )

# First Run
# ⏱️ 2.3 seconds
# 🌐 Live API Call
# 💾 Recorded

# Every Run After
# ⏱️ 12 ms
# ❌ No API Calls
# 📼 Replayed Locally

Why Sequa?

Supported Frameworks

• ✅ OpenAI
• ✅ Anthropic
• ✅ LangChain
• ✅ LangGraph

Installation

pip install sequa

or

uv add sequa

Quick Start

from langchain_groq import ChatGroq
from sequa import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke("Hello Sequa!")

That's it.

The first execution records the response.

Every matching execution after that replays it locally without calling the LLM.

Features

• 🧪 Phase 2 AI Regression Testing Engine: Compare reference cassettes against new executions across 5 analysis dimensions: Prompt Diff, Tool Diff, Semantic Diff, Cost Diff, and Latency Diff.
• 🔎 Searchable AI Executions & Instant Replay: Find past executions with TF-IDF cosine similarity (sequa search "refund" --since yesterday) and generate local replay code (sequa replay <hash>).
• 📼 Record once, replay forever
• ⚡ Replay, Record, Auto, Live, and Regression execution modes
• 🧰 Tool Calling & Function Calling support
• 🌊 Streaming support (sync & async)
• 🔒 PII & Sensitive Information Masking
• 🛡️ NVIDIA NeMo Guardrails Integration
• 🧠 Deterministic request hashing
• 🎯 Custom ignored fields
• 🔧 Custom request normalizers
• 🗂️ File, Memory & PostgreSQL storage backends
• 🧹 CLI utilities (regression, diff, search, replay, log, stats, inspect, clean)

Searchable AI Executions & Instant Replay (v0.5.0)

When a bug is reported ("The AI gave the wrong answer yesterday"), Sequa makes it reproducible in seconds:

# 1. Search past executions by TF-IDF cosine similarity & relative time
sequa search "refund request failed" --since yesterday -i

# 2. Inspect target execution and get instant Python replay code
sequa replay 1cea06570793

# 3. View chronological Git-like execution history
sequa log --path cassettes -n 5

Or query programmatically via Python API:

from sequa import search_cassettes

# Search recorded executions by natural language & metadata
results = search_cassettes(query="refund request", since="yesterday", provider="openai")

for res in results:
    print(f"Match Score: {res.score:.4f} | Hash: {res.hash[:12]}")
    print(f"  Input:  {res.input_snippet}")
    print(f"  Output: {res.output_snippet}")

CLI Reference

Sequa includes a built-in CLI to search, inspect, format, and debug your recorded cassettes:

Cassette Execution Diffing (sequa diff) 🔍

Compare any two recorded cassettes by hash, ID, or file path to inspect prompt, model, parameter, or output differences:

# 1. Compare two cassette executions in terminal text mode
sequa diff 1cea06570793 4b93d6e3 --path cassettes

# 2. Export diff comparison report to GitHub Markdown
sequa diff 1cea06570793 4b93d6e3 -f markdown -o diff.md

# 3. Export standalone styled HTML diff report
sequa diff 1cea06570793 4b93d6e3 -f html -o diff.html

# 4. Interactive Diff from Search Results
sequa search "refund request" -i
# Enter result numbers to diff (e.g., '1,2' or 'diff 1 2'):

Phase 2: AI Regression Testing Engine 🧪

Replay testing validates that an app runs deterministically against recorded cassettes. Phase 2 Regression Testing compares an Old Cassette (reference run) against a New Execution (live call or updated model) across 5 distinct analysis dimensions:

$$\text{Old Cassette} \longrightarrow \text{New Execution} \longrightarrow \begin{cases} \text{1. 📝 Prompt Diff} \ \text{2. 🛠️ Tool Diff} \ \text{3. 🧠 Semantic Diff} \ \text{4. 💰 Cost Diff} \ \text{5. ⚡ Latency Diff} \end{cases}$$

Python API

from sequa import compare_executions, cassette

# Programmatic 5-dimension comparison
report = compare_executions("tests/cassettes/ref_run.json", "tests/cassettes/new_run.json")

print(f"Semantic Similarity: {report.semantic_diff.similarity_score * 100:.1f}%")
print(f"Token Delta: {report.cost_diff.token_delta['total']:+d} tokens")
print(f"Latency Delta: {report.latency_diff.delta_ms:+.1f} ms")

# Enforce CI assertion rules
report.assert_no_regression(
    similarity_threshold=0.85,
    allow_tool_changes=False,
    max_cost_increase_pct=15.0,
    max_latency_increase_pct=25.0,
)

# Inline cassette execution with mode="regression"
with cassette(path="tests/cassettes/support_flow.json", mode="regression") as cas:
    response = model.invoke("How do I request a refund?")
    reg_report = cas.regression_report
    print(reg_report.render_text())

CLI Command

# Run 5-dimension regression test comparing two cassette recordings
sequa regression reference_run.json new_run.json

# Fail CI build if semantic output drifts below 85% threshold
sequa regression reference_run.json new_run.json --fail-on-drift --threshold 0.85

# Export GitHub Markdown or HTML regression report
sequa regression old_hash new_hash -f markdown -o regression_report.md
sequa regression old_hash new_hash -f html -o regression_report.html

Pytest Integration 🧪

Sequa includes built-in Pytest support via the sequa plugin.

Markers & Fixtures

Use @pytest.mark.sequa (or alias @pytest.mark.cassette) or inject the sequa_cassette fixture:

import pytest
from langchain_groq import ChatGroq

@pytest.mark.sequa(mode="auto")
def test_llm_feature():
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Say hello")
    assert "hello" in response.content.lower()

def test_with_fixture(sequa_cassette):
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Hello world")

Pytest CLI Flags

GitHub Actions Integration 🐙

Run deterministic LLM snapshot tests in CI/CD with zero API costs using the official Sequa GitHub Action.

Quick Workflow Setup

Add .github/workflows/ai-tests.yml to your repository:

name: AI Snapshot Tests

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Sequa LLM Tests
        uses: thetechnoadvisor/sequa@v1
        with:
          mode: replay
          cassette-path: tests/cassettes
          post-summary: true

Action Options

Common Use Cases

🚀 Speed up AI integration tests

Run your test suite in milliseconds instead of waiting for repeated LLM calls.

💰 Reduce API costs

Replay previously recorded executions without paying for another API request.

🧪 Deterministic testing

Replay the exact same execution every time.

💻 Offline development

Develop and test AI applications without internet connectivity.

🐞 Reproduce bugs

Replay the exact LLM interaction that caused the issue.

Storage Backends

Sequa supports multiple storage backends.

• 📁 File Storage
• 🧠 In-Memory Storage
• 🐘 PostgreSQL Storage

Choose whichever fits your workflow.

Documentation

Comprehensive documentation is available at:

👉 https://sequa.thetechnoadvisor.com/docs

Contributing

Contributions are always welcome.

• ⭐ Star the repository
• 🐞 Report bugs
• 💡 Suggest new features
• 🔧 Open a Pull Request

License

MIT License.