Tinman OpenClaw Eval

Security evaluation harness for OpenClaw agents. Powered by Tinman.

Features

• 280+ attack probes (currently 288) across 12 categories
• Synthetic Gateway for isolated testing
• CI integration via SARIF, JUnit, and JSON outputs
• Baseline assertions for regression testing
• Real-time monitoring via Gateway WebSocket

Attack Categories

Run tinman-eval list-attacks to see exact counts by category.

Installation

pip install tinman-openclaw-eval

Or from source:

git clone https://github.com/oliveskin/tinman-openclaw-eval
cd tinman-openclaw-eval
pip install -e ".[dev]"

Quick Start

# Run all attacks (mock gateway)
tinman-eval run

# Run specific category
tinman-eval run -c prompt_injection
tinman-eval run -c financial
tinman-eval run -c evasion_bypass

# Run only high severity (S3+)
tinman-eval run -s S3

# Save report
tinman-eval run -o report.md

# List all attacks
tinman-eval list-attacks

# Run single attack
tinman-eval run-single PI-001 -v

Category aliases are supported (e.g. financial, mcp_attacks, supplychain, platform).

CI Integration

GitHub Actions

name: Security Eval
on: [push, pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: pip install tinman-openclaw-eval

      - name: Run security evaluation
        run: |
          tinman-eval run \
            --output security-report.json \
            --format json

      - name: Assert baseline
        run: |
          tinman-eval assert-cmd \
            security-report.json \
            --baseline expected/baseline.json

      - name: Upload SARIF
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: security-report.sarif
        if: always()

      - name: Generate SARIF (always)
        run: |
          tinman-eval run \
            --output security-report.sarif \
            --format sarif
        if: always()

Generate Baseline

# Create initial baseline
tinman-eval baseline --output expected/baseline.json

# Update after intentional changes
tinman-eval run -f json -o new-results.json
# Review and approve
mv new-results.json expected/baseline.json

Output Formats

Custom Attacks

Add a new attack module under src/tinman_openclaw_eval/attacks/:

from tinman_openclaw_eval.attacks.base import Attack, AttackCategory, AttackPayload, ExpectedBehavior, Severity


class MyAttacks(Attack):
    category = AttackCategory.PROMPT_INJECTION
    name = "My Attacks"

    def _load_payloads(self) -> None:
        self.payloads.append(
            AttackPayload(
                id="MY-001",
                name="My probe",
                category=self.category,
                severity=Severity.S2,
                payload="...",
                target="dm_channel",
                expected_behavior=ExpectedBehavior.REJECTED_BY_SOUL,
            )
        )

Then export it from src/tinman_openclaw_eval/attacks/__init__.py and register it in src/tinman_openclaw_eval/harness.py.

Programmatic Usage

import asyncio
from tinman_openclaw_eval import EvalHarness, AttackCategory

async def main():
    harness = EvalHarness()

    # Run all attacks
    result = await harness.run()

    # Check for vulnerabilities
    print(f"Vulnerabilities: {result.vulnerabilities}")

    # Run specific categories
    result = await harness.run(categories=[
        AttackCategory.PROMPT_INJECTION,
        AttackCategory.FINANCIAL_TRANSACTION,
        AttackCategory.EVASION_BYPASS,
    ])

    # Run high severity only
    result = await harness.run(min_severity="S3")

asyncio.run(main())

Testing Against Real Gateway

# Connect to local OpenClaw Gateway
tinman-eval run --no-mock --gateway-url ws://127.0.0.1:18789

# With custom config
tinman-eval run --no-mock --gateway-url ws://192.168.1.100:18789

Attack Probe IDs

Severity Levels

Integration with OpenClaw Skill

For continuous monitoring in OpenClaw, use the Tinman Skill:

# In OpenClaw
/tinman sweep                    # Run security sweep
/tinman sweep --category financial
/tinman watch                    # Real-time monitoring

Links

• Tinman - AI Failure Mode Research
• Tinman Skill - OpenClaw Integration
• OpenClaw - Personal AI Assistant

License

Apache-2.0