Icho 📼

Deterministic testing for AI applications.

Record once. Replay forever.

---

Stop paying for every AI test run.

Every time your AI application runs during testing, it probably:

• 💸 Calls the LLM again
• 🐢 Slows down your CI pipeline
• 🎲 Produces slightly different outputs
• 🌐 Depends on internet connectivity

Icho records a real AI execution once and replays it locally during future test runs.

The result

• ⚡ Millisecond replay
• 💰 Zero replay API costs
• 🧪 Deterministic testing
• 💻 Works offline

---

Before

from langchain_groq import ChatGroq

model = ChatGroq(model_name="llama-3.1-8b-instant")

response = model.invoke(
    "Write a 3-word slogan for gravity."
)

# ⏱️ 2.3 seconds
# 🌐 Live API Call

---

After

from langchain_groq import ChatGroq
from icho import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke(
        "Write a 3-word slogan for gravity."
    )

# First Run
# ⏱️ 2.3 seconds
# 🌐 Live API Call
# 💾 Recorded

# Every Run After
# ⏱️ 12 ms
# ❌ No API Calls
# 📼 Replayed Locally

---

Why Icho?

Without Icho	With Icho
Calls the LLM on every test	Record once, replay forever
Seconds of latency	Millisecond replay
API cost every execution	No replay API cost
Internet required	Works offline
Non-deterministic	Deterministic

---

Supported Frameworks

• ✅ OpenAI
• ✅ Anthropic
• ✅ LangChain
• ✅ LangGraph

---

Installation

pip install icho

or

uv add icho

---

Quick Start

from langchain_groq import ChatGroq
from icho import cassette

model = ChatGroq(model_name="llama-3.1-8b-instant")

with cassette("tests/cassettes"):
    response = model.invoke("Hello Icho!")

That's it.

The first execution records the response.

Every matching execution after that replays it locally without calling the LLM.

---

Features

• 🧪 Phase 2 AI Regression Testing Engine: Compare reference cassettes against new executions across 5 analysis dimensions: Prompt Diff, Tool Diff, Semantic Diff, Cost Diff, and Latency Diff.
• 🔎 Searchable AI Executions & Instant Replay: Find past executions with TF-IDF cosine similarity (icho search "refund" --since yesterday) and generate local replay code (icho replay <hash>).
• 📼 Record once, replay forever
• ⚡ Replay, Record, Auto, Live, and Regression execution modes
• 🧰 Tool Calling & Function Calling support
• 🌊 Streaming support (sync & async)
• 🔒 PII & Sensitive Information Masking
• 🛡️ NVIDIA NeMo Guardrails Integration
• 🧠 Deterministic request hashing
• 🎯 Custom ignored fields
• 🔧 Custom request normalizers
• 🗂️ File, Memory & PostgreSQL storage backends
• 🧹 CLI utilities: Record (icho record python app.py), Replay (icho replay python app.py), Diff (icho diff), Test (icho test), Eval (icho eval), search, log, stats, inspect, clean

---

Killer CLI Workflow ⚡

Icho makes testing and evaluating AI applications as simple as 4 core commands:

Developer runs AI application
          ↓
     icho record python app.py
          ↓
     ┌──────────────┐
     │   Cassette   │
     └──────────────┘
          ↓
 ┌────────┬────────┬────────┬────────┐
  Replay    Diff     Test     Eval

1. icho record

Run your AI application command — icho intercepts and records all LLM requests/responses to disk without altering a single line of your code:

icho record python app.py

2. icho replay

Run your AI application in 100% offline, zero-token-cost replay mode, or inspect single cassettes:

# Run application offline with 0 API costs
icho replay python app.py

# Inspect cassette replay details & code snippet
icho replay <cassette_hash>

3. icho diff

Compare cassette executions automatically (defaults to the 2 most recent recordings) or perform 5-dimension Phase 2 regression diff analysis:

# Compare 2 latest recordings
icho diff

# Run 5-dimension Phase 2 regression analysis (Prompt, Tool, Semantic, Cost, Latency)
icho diff execution_1 execution_2 --regression

4. icho test

Execute batch snapshot test suites across directory cassettes in parallel:

icho test tests/cassettes --mode replay

5. icho eval

Evaluate recorded cassettes for NeMo guardrail compliance, PII leakages, JSON schema accuracy, and quality metrics:

icho eval cassettes/

---

Searchable AI Executions & Instant Replay

When a bug is reported ("The AI gave the wrong answer yesterday"), Icho makes it reproducible in seconds:

# 1. Search past executions by TF-IDF cosine similarity & relative time
icho search "refund request failed" --since yesterday -i

# 2. Inspect target execution and get instant Python replay code
icho replay 1cea06570793

# 3. View chronological Git-like execution history
icho log --path cassettes -n 5

Or query programmatically via Python API:

from icho import search_cassettes

# Search recorded executions by natural language & metadata
results = search_cassettes(query="refund request", since="yesterday", provider="openai")

for res in results:
    print(f"Match Score: {res.score:.4f} | Hash: {res.hash[:12]}")
    print(f"  Input:  {res.input_snippet}")
    print(f"  Output: {res.output_snippet}")

---

CLI Reference

Icho includes a built-in CLI to search, inspect, format, and debug your recorded cassettes:

Command	Description	Example
icho diff	Compare two cassette executions (text, markdown, html formats)	icho diff hash1 hash2 -f html -o diff.html
icho search	Search executions by vector cosine similarity, time window, or provider/model	icho search "refund" --since yesterday -i
icho log	Show Git-like chronological execution log	icho log --path cassettes -n 10
icho replay	Inspect a target cassette and generate copy-paste Python replay code	icho replay 1cea06570793
icho stats	View total cassette count, disk size, and saved API latency	icho stats --path cassettes
icho inspect	List all saved cassettes with providers, models, and timestamps	icho inspect --path cassettes
icho clean	Redact volatile timestamps and latency before committing to Git	icho clean --remove-latency --remove-timestamps

---

Cassette Execution Diffing (icho diff) 🔍

Compare any two recorded cassettes by hash, ID, or file path to inspect prompt, model, parameter, or output differences:

# 1. Compare two cassette executions in terminal text mode
icho diff 1cea06570793 4b93d6e3 --path cassettes

# 2. Export diff comparison report to GitHub Markdown
icho diff 1cea06570793 4b93d6e3 -f markdown -o diff.md

# 3. Export standalone styled HTML diff report
icho diff 1cea06570793 4b93d6e3 -f html -o diff.html

# 4. Interactive Diff from Search Results
icho search "refund request" -i
# Enter result numbers to diff (e.g., '1,2' or 'diff 1 2'):

---

Phase 2: AI Regression Testing Engine 🧪

Replay testing validates that an app runs deterministically against recorded cassettes. Phase 2 Regression Testing compares an Old Cassette (reference run) against a New Execution (live call or updated model) across 5 distinct analysis dimensions:

$$\text{Old Cassette} \longrightarrow \text{New Execution} \longrightarrow \begin{cases} \text{1. 📝 Prompt Diff} \ \text{2. 🛠️ Tool Diff} \ \text{3. 🧠 Semantic Diff} \ \text{4. 💰 Cost Diff} \ \text{5. ⚡ Latency Diff} \end{cases}$$

Python API

from icho import compare_executions, cassette

# Programmatic 5-dimension comparison
report = compare_executions("tests/cassettes/ref_run.json", "tests/cassettes/new_run.json")

print(f"Semantic Similarity: {report.semantic_diff.similarity_score * 100:.1f}%")
print(f"Token Delta: {report.cost_diff.token_delta['total']:+d} tokens")
print(f"Latency Delta: {report.latency_diff.delta_ms:+.1f} ms")

# Enforce CI assertion rules
report.assert_no_regression(
    similarity_threshold=0.85,
    allow_tool_changes=False,
    max_cost_increase_pct=15.0,
    max_latency_increase_pct=25.0,
)

# Inline cassette execution with mode="regression"
with cassette(path="tests/cassettes/support_flow.json", mode="regression") as cas:
    response = model.invoke("How do I request a refund?")
    reg_report = cas.regression_report
    print(reg_report.render_text())

CLI Command

# Run 5-dimension regression test comparing two cassette recordings
icho regression reference_run.json new_run.json

# Fail CI build if semantic output drifts below 85% threshold
icho regression reference_run.json new_run.json --fail-on-drift --threshold 0.85

# Export GitHub Markdown or HTML regression report
icho regression old_hash new_hash -f markdown -o regression_report.md
icho regression old_hash new_hash -f html -o regression_report.html

---

AI Test Datasets (icho test) 🧪

Organize thousands of conversation scenarios into domain directories using .yaml or .json cassettes:

tests/
├── customer_support/
│   ├── greeting.yaml
│   ├── refund.yaml
│   └── complaint.yaml
├── travel_agent/
│   ├── booking.yaml
│   └── cancellation.yaml
└── finance/
    └── invoices.yaml

CLI Command

# Run batch test suite across a domain directory
icho test tests/customer_support/

# Run multi-domain dataset with parallel worker pool
icho test tests/ -j 4

# Run regression suite across datasets against reference cassettes
icho test tests/customer_support/ --mode regression --threshold 0.85 --fail-on-drift

Python API

from icho import discover_suite, SuiteRunner

# Discover nested test suite hierarchy
suite = discover_suite("tests/customer_support")

# Execute batch test suite
runner = SuiteRunner(mode="replay")
report = runner.run(suite)

print(report.render_text())

---

Pytest Integration 🧪

Icho includes built-in Pytest support via the icho plugin.

Markers & Fixtures

Use @pytest.mark.icho (or alias @pytest.mark.sequa / @pytest.mark.cassette) or inject the icho_cassette fixture:

import pytest
from langchain_groq import ChatGroq

@pytest.mark.icho(mode="auto")
def test_llm_feature():
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Say hello")
    assert "hello" in response.content.lower()

def test_with_fixture(icho_cassette):
    model = ChatGroq(model_name="llama-3.1-8b-instant")
    response = model.invoke("Hello world")

Pytest CLI Flags

Flag	Description	Default
--icho-mode=<mode>	Globally override cassette mode (auto, record, replay, live)	Marker / auto
--icho-path=<path>	Set base cassette directory	tests/cassettes
--icho-mask-pii	Enable PII masking across all test cassette recordings	False

---

GitHub Actions Integration 🐙

Run deterministic LLM snapshot tests in CI/CD with zero API costs using the official Icho GitHub Action.

Quick Workflow Setup

Add .github/workflows/ai-tests.yml to your repository:

name: AI Snapshot Tests

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Icho LLM Tests
        uses: thetechnoadvisor/icho@main
        with:
          mode: replay
          cassette-path: tests/cassettes
          post-summary: true

Action Options

Input	Description	Default
mode	Execution mode (replay, auto, record, live)	replay
cassette-path	Path to saved cassette files directory	tests/cassettes
python-version	Python version for setup	3.12
pytest-args	Additional arguments passed to pytest	""
post-summary	Post cassette stats report to $GITHUB_STEP_SUMMARY	true

---

Common Use Cases

🚀 Speed up AI integration tests

Run your test suite in milliseconds instead of waiting for repeated LLM calls.

---

💰 Reduce API costs

Replay previously recorded executions without paying for another API request.

---

🧪 Deterministic testing

Replay the exact same execution every time.

---

💻 Offline development

Develop and test AI applications without internet connectivity.

---

🐞 Reproduce bugs

Replay the exact LLM interaction that caused the issue.

---

Storage Backends

Icho supports multiple storage backends:

• 📁 File Storage
• 🧠 In-Memory Storage
• 🐘 PostgreSQL Storage

Choose whichever fits your workflow.

---

Contributing

Contributions are always welcome.

• ⭐ Star the repository
• 🐞 Report bugs
• 💡 Suggest new features
• 🔧 Open a Pull Request

---

License

MIT License.