AgentTinman 0.2.1

Forward-Deployed Research Agent for AI system failure discovery

Agent Tinman

Forward-Deployed Research Agent for Continuous AI Reliability Discovery

Quick Start  •   How It Works  •   API  •   Documentation

---

Tinman is not a testing tool. It's an autonomous research agent that continuously explores your AI system's behavior to discover failure modes you haven't imagined yet.

While traditional approaches wait for failures to happen, Tinman proactively generates hypotheses about what could fail, designs experiments to test them, and proposes interventions—all with human oversight at critical decision points.

Why Tinman?

The problem: Every team deploying LLMs faces the same question: "What don't we know about how this system can fail?"

Most tools help you monitor what you've already anticipated. Tinman helps you discover what you haven't.

Traditional Approach - Reactive—triggered after incidents - Tests known failure patterns - Output: pass/fail results - Goal: verify correctness - Stops when tests pass

Tinman + Proactive—always exploring + Generates novel hypotheses + Output: understanding + Goal: expand knowledge + Never stops—research is ongoing

Core Capabilities

Capability	Description
Hypothesis-Driven Research	Generates testable hypotheses about potential failure modes based on system architecture and observed behavior
Controlled Experimentation	Tests each hypothesis with configurable parameters, cost controls, and reproducibility tracking
Failure Classification	Classifies failures using a structured taxonomy with severity ratings (S0-S4)
Intervention Design	Proposes concrete fixes: prompt mutations, guardrails, tool policy changes, architectural recommendations
Simulation & Validation	Validates interventions through counterfactual replay before deployment
Human-in-the-Loop	Risk-tiered approval gates ensure humans control consequential decisions

Quick Start

Installation

pip install AgentTinman

With specific model provider support:

pip install AgentTinman[openai]     # OpenAI
pip install AgentTinman[anthropic]  # Anthropic
pip install AgentTinman[all]        # All providers

Initialize & Run

# Initialize configuration
tinman init

# Launch the interactive TUI
tinman tui

# Or run a research cycle directly
tinman research --focus "tool use failures"

# Generate a report
tinman report --format markdown

Configure Your Model

Edit .tinman/config.yaml:

mode: lab

models:
  default: openai
  providers:
    openai:
      api_key: ${OPENAI_API_KEY}
      model: gpt-4-turbo-preview
    groq:
      api_key: ${GROQ_API_KEY}
      model: llama3-70b-8192

Set API keys as environment variables:

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export GROQ_API_KEY="..."

Supported Providers

Provider	Cost	Best For
Ollama	Free (local)	Privacy, offline, unlimited experimentation
Groq	Free tier	Speed, high volume
OpenRouter	Many free models	Variety—DeepSeek, Qwen, Llama, Mistral
Together	$25 free credits	Quality open models
OpenAI	Paid	GPT-4
Anthropic	Paid	Claude

The Research Cycle

┌─────────────────────────────────────────────────────────────────────┐
│                         RESEARCH CYCLE                              │
│                                                                     │
│   ┌────────────┐    ┌────────────┐    ┌────────────┐               │
│   │ Hypothesis │───▶│ Experiment │───▶│  Failure   │               │
│   │   Engine   │    │  Executor  │    │ Discovery  │               │
│   └────────────┘    └────────────┘    └─────┬──────┘               │
│         ▲                                   │                       │
│         │           ┌────────────┐    ┌─────▼──────┐               │
│         │           │ Simulation │◀───│Intervention│               │
│         │           │   Engine   │    │   Engine   │               │
│         │           └─────┬──────┘    └────────────┘               │
│         │                 │                                         │
│         └─────── Learning ◀┘                                        │
│                (Memory Graph)                                       │
└─────────────────────────────────────────────────────────────────────┘

Each cycle:

1. Generate hypotheses about potential failures
2. Design experiments to test each hypothesis
3. Execute experiments with approval gates
4. Discover and classify failures found
5. Design interventions to address failures
6. Simulate fixes via counterfactual replay
7. Learn from results for future cycles

Operating Modes

Tinman operates in three modes with different safety boundaries:

Mode	Purpose	Approval Gates	Destructive Tests
LAB	Unrestricted research	Auto-approve most	Allowed
SHADOW	Observe production traffic	Review S3+ severity	Read-only
PRODUCTION	Active protection	Human approval required	Blocked

Transition rules:

• LAB → SHADOW → PRODUCTION (progressive rollout)
• PRODUCTION → SHADOW (regression fallback)
• Cannot skip modes (LAB → PRODUCTION blocked)

Failure Taxonomy

Tinman classifies failures into six primary classes:

Class	Description	Example
REASONING	Logical errors, goal drift, hallucination	Model contradicts itself mid-response
LONG_CONTEXT	Context window issues, attention dilution	Forgets instructions from early in conversation
TOOL_USE	Tool call failures, parameter errors, exfil	Calls API with invalid arguments
FEEDBACK_LOOP	Output amplification, error cascades	Retry loop amplifies initial mistake
DEPLOYMENT	Infrastructure, latency, resource issues	Timeout under load causes partial response
SECURITY	Credential access, data exfil, evasion, injection	Agent attempts to read SSH keys or bypasses filters

Security Failure Types:

Type	Description
credential_access	Attempted access to SSH keys, API tokens, wallets
data_exfiltration	Sending sensitive data to external locations
unauthorized_action	Taking actions without explicit user consent
privilege_escalation	Attempting sudo, sandbox bypass, elevation
injection_susceptible	Following injected instructions from untrusted content
evasion_bypass	Using encoding tricks to bypass security controls
memory_poisoning	Injecting malicious context into agent memory
platform_specific_attack	OS-specific attacks (mimikatz, LaunchAgents, systemd)

Severity levels:

Level	Impact	Action
S0	Benign	Monitor
S1	UX degradation	Review
S2	Business risk	Investigate
S3	Serious risk	Mitigate
S4	Critical	Immediate action

Human-in-the-Loop Approval

Risk-tiered approval balances autonomy with safety:

Action Request
      │
      ▼
┌─────────────┐
│    Risk     │
│  Evaluator  │
└──────┬──────┘
       │
   ┌───┴───┐
   │       │
   ▼       ▼
 SAFE    REVIEW ───▶ Human Decision ───▶ Approved/Rejected
   │       │
   │       ▼
   │     BLOCK ───▶ Rejected (always)
   ▼
Auto-Approved

Risk Tiers:

• SAFE: Low-risk actions proceed automatically
• REVIEW: Medium-risk actions require human approval
• BLOCK: High-risk actions are always rejected

Python API

Basic Usage

import asyncio
from tinman import create_tinman
from tinman.config.modes import Mode

async def main():
    tinman = await create_tinman(
        mode=Mode.LAB,
        db_url="sqlite:///tinman.db"
    )

    results = await tinman.research_cycle(
        focus="reasoning failures in multi-step tasks",
        max_hypotheses=5,
        max_experiments=3
    )

    print(f"Hypotheses tested: {len(results.hypotheses)}")
    print(f"Failures discovered: {len(results.failures)}")
    print(f"Interventions proposed: {len(results.interventions)}")

    report = await tinman.generate_report(format="markdown")
    print(report)

    await tinman.close()

asyncio.run(main())

Pipeline Integration

from tinman.integrations import PipelineAdapter
from tinman.config.modes import Mode

adapter = PipelineAdapter(mode=Mode.SHADOW)

async def monitored_llm_call(messages):
    ctx = adapter.create_context(messages=messages)
    ctx = await adapter.pre_request(ctx)

    response = await your_existing_llm_call(messages)

    ctx.response = response
    ctx = await adapter.post_request(ctx)
    return response

Real-Time Gateway Monitoring

Connect to AI gateway WebSockets for instant event analysis:

from tinman.integrations.gateway_plugin import GatewayMonitor, ConsoleAlerter, FileAlerter

monitor = GatewayMonitor(your_adapter)
monitor.add_alerter(ConsoleAlerter())
monitor.add_alerter(FileAlerter("~/tinman-findings.md"))
await monitor.start()

Platform adapters:

• OpenClaw — Security eval harness + gateway adapter

Configuration Reference

# .tinman/config.yaml

mode: lab  # lab, shadow, or production

database:
  url: sqlite:///tinman.db
  pool_size: 10

models:
  default: openai
  providers:
    openai:
      api_key: ${OPENAI_API_KEY}
      model: gpt-4-turbo-preview
      temperature: 0.7

research:
  max_hypotheses_per_run: 10
  max_experiments_per_hypothesis: 3
  default_runs_per_experiment: 5

experiments:
  max_parallel: 5
  default_timeout_seconds: 300
  cost_limit_usd: 10.0

risk:
  auto_approve_safe: true
  block_on_destructive: true

approval:
  mode: interactive  # interactive, async, auto_approve, auto_reject
  timeout_seconds: 300

Architecture

tinman/
├── agents/                 # Autonomous research agents
│   ├── hypothesis_engine.py
│   ├── experiment_architect.py
│   ├── experiment_executor.py
│   ├── failure_discovery.py
│   ├── intervention_engine.py
│   └── simulation_engine.py
├── config/                 # Configuration and modes
│   ├── modes.py            # LAB/SHADOW/PRODUCTION
│   └── settings.py
├── core/                   # Infrastructure
│   ├── approval_gate.py
│   ├── control_plane.py
│   ├── risk_policy.py
│   ├── cost_tracker.py
│   └── tools.py
├── db/                     # Persistence
│   ├── connection.py
│   ├── models.py
│   └── audit.py
├── integrations/           # External connections
│   ├── model_client.py
│   ├── pipeline_adapter.py
│   ├── gateway_plugin/     # Real-time monitoring
│   └── *_client.py         # Provider clients
├── memory/                 # Knowledge graph
│   ├── graph.py
│   ├── models.py
│   └── repository.py
├── taxonomy/               # Failure classification
│   ├── failure_types.py
│   └── classifiers.py
├── reporting/              # Report generation
└── tui/                    # Terminal UI

Documentation

Document	Description
QUICKSTART.md	Get running in 5 minutes
CONCEPTS.md	Core mental model and abstractions
ARCHITECTURE.md	System design and data flow
TAXONOMY.md	Complete failure classification guide
MODES.md	Operating mode behavior matrix
HITL.md	Human-in-the-loop approval system
INTEGRATION.md	Pipeline integration patterns
CONFIGURATION.md	All configuration options

Live docs: oliveskin.github.io/Agent-Tinman

Use Cases

Research Lab — Run in LAB mode against development deployments to discover failures before production.

Shadow Monitoring — Deploy in SHADOW mode alongside production to observe real traffic and surface emergent failure modes.

Production Protection — Run in PRODUCTION mode with human approval gates to actively protect against discovered patterns.

Compliance & Audit — Use the memory graph to demonstrate due diligence: what was discovered, what was applied, what the outcomes were.

Real-Time Gateway Monitoring — Connect to AI gateway WebSockets for instant event analysis as failures happen.

Philosophy

Tinman embodies a research methodology, not just a tool:

1. Systematic curiosity — Continuously ask "what could go wrong?" rather than "does this work?"
2. Hypothesis-driven — Every test has a reason. No random fuzzing.
3. Human oversight — Autonomy where safe, human judgment where it matters.
4. Temporal knowledge — Not just "what failed" but "what did we know, when?"
5. Continuous learning — Each cycle informs the next. Knowledge compounds.

Contributing

We welcome contributions. See CONTRIBUTING.md for:

• Development setup
• Code style (ruff, mypy)
• Testing requirements
• PR process

License

Apache 2.0 — See LICENSE

---

Tinman is a public good.
Not monetized, not proprietary—just a crystallized methodology for systematic AI reliability research.

GitHub  •   PyPI  •   Docs