testthread 0.5.0

pytest for AI agents

TestThread 🧵

pytest for AI agents.

The open-source testing framework that tells you if your AI agent is actually working — or quietly breaking.

---

The Problem

You build an AI agent. It works in testing. You ship it.

Then it starts hallucinating. Returning wrong formats. Calling the wrong tools. Breaking your pipeline.

You find out when something downstream crashes — not before.

TestThread fixes that.

---

What TestThread Does

Define what your agent should do. TestThread runs it, checks the output, and tells you exactly what passed and what failed.

• ✅ Define test suites per agent
• ✅ Add test cases with expected outputs
• ✅ Run suites against your live agent endpoint
• ✅ Get pass/fail results with reasons
• ✅ Track pass rate over time
• ✅ Catch regressions before they hit production

---

Quick Start

pip install requests

import requests

BASE = "https://test-thread-production.up.railway.app"

# Create a test suite
suite = requests.post(f"{BASE}/suites", json={
    "name": "My Agent Tests",
    "description": "Testing my AI agent",
    "agent_endpoint": "https://your-agent.com/run"
}).json()

# Add a test case
requests.post(f"{BASE}/suites/{suite['id']}/cases", json={
    "name": "Basic response check",
    "input": "What is 2 + 2?",
    "expected_output": "4",
    "match_type": "contains"
})

# Run the suite
result = requests.post(f"{BASE}/suites/{suite['id']}/run").json()
print(f"Passed: {result['passed']} | Failed: {result['failed']}")

---

Match Types

Type	Description
contains	Output contains the expected string
exact	Output matches exactly
regex	Output matches a regex pattern

---

Live Dashboard

View your test results visually at test-thread.lovable.app

---

API Reference

Full docs at test-thread-production.up.railway.app/docs

Method	Endpoint	Description
GET	/	Health check
POST	/suites	Create test suite
GET	/suites	List all suites
POST	/suites/{id}/cases	Add test case
GET	/suites/{id}/cases	List test cases
POST	/suites/{id}/run	Run suite
GET	/runs	List all runs
GET	/runs/{id}	Get run details
GET	/dashboard/stats	Dashboard stats

---

Part of the Thread Suite

TestThread is part of a suite of open-source reliability tools for AI agents.

Tool	What it does
Iron-Thread	Validates AI output structure before it hits your database
TestThread	Tests whether your agent behaves correctly across runs
PromptThread (coming soon)	Versions and tracks prompt performance over time

---

Self-Host

git clone https://github.com/eugene001dayne/test-thread.git
cd test-thread
pip install -r requirements.txt
uvicorn main:app --reload

---

License

Apache 2.0 — free to use, modify, and distribute.

---

Built for developers who ship AI agents and need to know they work.