Agent Tinman

Forward-Deployed Research Agent for Continuous AI Reliability Discovery

Quick Start  •   How It Works  •   API  •   Documentation

Tinman is not a testing tool. It's an autonomous research agent that continuously explores your AI system's behavior to discover failure modes you haven't imagined yet.

While traditional approaches wait for failures to happen, Tinman proactively generates hypotheses about what could fail, designs experiments to test them, and proposes interventions—all with human oversight at critical decision points.

Why Tinman?

The problem: Every team deploying LLMs faces the same question: "What don't we know about how this system can fail?"

Most tools help you monitor what you've already anticipated. Tinman helps you discover what you haven't.

- Reactive—triggered after incidents
- Tests known failure patterns
- Output: pass/fail results
- Goal: verify correctness
- Stops when tests pass

+ Proactive—always exploring
+ Generates novel hypotheses
+ Output: understanding
+ Goal: expand knowledge
+ Never stops—research is ongoing

Core Capabilities

Quick Start

Installation

pip install AgentTinman

With specific model provider support:

pip install AgentTinman[openai]     # OpenAI
pip install AgentTinman[anthropic]  # Anthropic
pip install AgentTinman[all]        # All providers

Initialize & Run

# Initialize configuration
tinman init

# Launch the interactive TUI
tinman tui

# Or run a research cycle directly
tinman research --focus "tool use failures"

# Generate a report
tinman report --format markdown

Configure Your Model

Edit .tinman/config.yaml:

mode: lab

models:
  default: openai
  providers:
    openai:
      api_key: ${OPENAI_API_KEY}
      model: gpt-4-turbo-preview
    groq:
      api_key: ${GROQ_API_KEY}
      model: llama3-70b-8192

Set API keys as environment variables:

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export GROQ_API_KEY="..."

Supported Providers

The Research Cycle

┌─────────────────────────────────────────────────────────────────────┐
│                         RESEARCH CYCLE                              │
│                                                                     │
│   ┌────────────┐    ┌────────────┐    ┌────────────┐               │
│   │ Hypothesis │───▶│ Experiment │───▶│  Failure   │               │
│   │   Engine   │    │  Executor  │    │ Discovery  │               │
│   └────────────┘    └────────────┘    └─────┬──────┘               │
│         ▲                                   │                       │
│         │           ┌────────────┐    ┌─────▼──────┐               │
│         │           │ Simulation │◀───│Intervention│               │
│         │           │   Engine   │    │   Engine   │               │
│         │           └─────┬──────┘    └────────────┘               │
│         │                 │                                         │
│         └─────── Learning ◀┘                                        │
│                (Memory Graph)                                       │
└─────────────────────────────────────────────────────────────────────┘

Each cycle:

1. Generate hypotheses about potential failures
2. Design experiments to test each hypothesis
3. Execute experiments with approval gates
4. Discover and classify failures found
5. Design interventions to address failures
6. Simulate fixes via counterfactual replay
7. Learn from results for future cycles

Operating Modes

Tinman operates in three modes with different safety boundaries:

Transition rules:

• LAB → SHADOW → PRODUCTION (progressive rollout)
• PRODUCTION → SHADOW (regression fallback)
• Cannot skip modes (LAB → PRODUCTION blocked)

Failure Taxonomy

Tinman classifies failures into six primary classes:

Security Failure Types:

Severity levels:

Human-in-the-Loop Approval

Risk-tiered approval balances autonomy with safety:

Action Request
      │
      ▼
┌─────────────┐
│    Risk     │
│  Evaluator  │
└──────┬──────┘
       │
   ┌───┴───┐
   │       │
   ▼       ▼
 SAFE    REVIEW ───▶ Human Decision ───▶ Approved/Rejected
   │       │
   │       ▼
   │     BLOCK ───▶ Rejected (always)
   ▼
Auto-Approved

Risk Tiers:

• SAFE: Low-risk actions proceed automatically
• REVIEW: Medium-risk actions require human approval
• BLOCK: High-risk actions are always rejected

Python API

Basic Usage

import asyncio
from tinman import create_tinman
from tinman.config.modes import Mode

async def main():
    tinman = await create_tinman(
        mode=Mode.LAB,
        db_url="sqlite:///tinman.db"
    )

    results = await tinman.research_cycle(
        focus="reasoning failures in multi-step tasks",
        max_hypotheses=5,
        max_experiments=3
    )

    print(f"Hypotheses tested: {len(results.hypotheses)}")
    print(f"Failures discovered: {len(results.failures)}")
    print(f"Interventions proposed: {len(results.interventions)}")

    report = await tinman.generate_report(format="markdown")
    print(report)

    await tinman.close()

asyncio.run(main())

Pipeline Integration

from tinman.integrations import PipelineAdapter
from tinman.config.modes import Mode

adapter = PipelineAdapter(mode=Mode.SHADOW)

async def monitored_llm_call(messages):
    ctx = adapter.create_context(messages=messages)
    ctx = await adapter.pre_request(ctx)

    response = await your_existing_llm_call(messages)

    ctx.response = response
    ctx = await adapter.post_request(ctx)
    return response

Real-Time Gateway Monitoring

Connect to AI gateway WebSockets for instant event analysis:

from tinman.integrations.gateway_plugin import GatewayMonitor, ConsoleAlerter, FileAlerter

monitor = GatewayMonitor(your_adapter)
monitor.add_alerter(ConsoleAlerter())
monitor.add_alerter(FileAlerter("~/tinman-findings.md"))
await monitor.start()

Platform adapters:

• OpenClaw — Security eval harness + gateway adapter

Configuration Reference

# .tinman/config.yaml

mode: lab  # lab, shadow, or production

database:
  url: sqlite:///tinman.db
  pool_size: 10

models:
  default: openai
  providers:
    openai:
      api_key: ${OPENAI_API_KEY}
      model: gpt-4-turbo-preview
      temperature: 0.7

research:
  max_hypotheses_per_run: 10
  max_experiments_per_hypothesis: 3
  default_runs_per_experiment: 5

experiments:
  max_parallel: 5
  default_timeout_seconds: 300
  cost_limit_usd: 10.0

risk:
  auto_approve_safe: true
  block_on_destructive: true

approval:
  mode: interactive  # interactive, async, auto_approve, auto_reject
  timeout_seconds: 300

Architecture

tinman/
├── agents/                 # Autonomous research agents
│   ├── hypothesis_engine.py
│   ├── experiment_architect.py
│   ├── experiment_executor.py
│   ├── failure_discovery.py
│   ├── intervention_engine.py
│   └── simulation_engine.py
├── config/                 # Configuration and modes
│   ├── modes.py            # LAB/SHADOW/PRODUCTION
│   └── settings.py
├── core/                   # Infrastructure
│   ├── approval_gate.py
│   ├── control_plane.py
│   ├── risk_policy.py
│   ├── cost_tracker.py
│   └── tools.py
├── db/                     # Persistence
│   ├── connection.py
│   ├── models.py
│   └── audit.py
├── integrations/           # External connections
│   ├── model_client.py
│   ├── pipeline_adapter.py
│   ├── gateway_plugin/     # Real-time monitoring
│   └── *_client.py         # Provider clients
├── memory/                 # Knowledge graph
│   ├── graph.py
│   ├── models.py
│   └── repository.py
├── taxonomy/               # Failure classification
│   ├── failure_types.py
│   └── classifiers.py
├── reporting/              # Report generation
└── tui/                    # Terminal UI

Documentation

Live docs: oliveskin.github.io/Agent-Tinman

Use Cases

Research Lab — Run in LAB mode against development deployments to discover failures before production.

Shadow Monitoring — Deploy in SHADOW mode alongside production to observe real traffic and surface emergent failure modes.

Production Protection — Run in PRODUCTION mode with human approval gates to actively protect against discovered patterns.

Compliance & Audit — Use the memory graph to demonstrate due diligence: what was discovered, what was applied, what the outcomes were.

Real-Time Gateway Monitoring — Connect to AI gateway WebSockets for instant event analysis as failures happen.

Philosophy

Tinman embodies a research methodology, not just a tool:

1. Systematic curiosity — Continuously ask "what could go wrong?" rather than "does this work?"
2. Hypothesis-driven — Every test has a reason. No random fuzzing.
3. Human oversight — Autonomy where safe, human judgment where it matters.
4. Temporal knowledge — Not just "what failed" but "what did we know, when?"
5. Continuous learning — Each cycle informs the next. Knowledge compounds.

Contributing

We welcome contributions. See CONTRIBUTING.md for:

• Development setup
• Code style (ruff, mypy)
• Testing requirements
• PR process

License

Apache 2.0 — See LICENSE

Tinman is a public good.
Not monetized, not proprietary—just a crystallized methodology for systematic AI reliability research.

GitHub  •   PyPI  •   Docs